Reliability and Security in the
CoDeeN
Content Distribution Network
Limin Wang, KyoungSoo Park, Ruoming Pang,
Vivek Pai, Larry Peterson
Princeton University
What Is CoDeeN?
Academic Content Distribution Network
Forward/reverse proxies, redirector
100+ proxy servers on PlanetLab
Continuous service, decentralized
control
Deployed for getting real traffic
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
2
Goals of CoDeeN
Provide open content distribution
Improve web performance & reliability
Platform for testing new innovations
Particularly in live environments
Keep CoDeeN running 24/7
Security
Reliability
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
3
How Does CoDeeN Work?
origin
Request
Cache hit
CoDeeN
Proxy
Cache hit
Each CoDeeN proxy is a forward
proxy, reverse proxy, & redirector
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
4
By The Numbers…
Running 24/7 since June 2003*
Over 870,000 unique IPs as clients
Over 500 million requests serviced
Valid rates up to 400K reqs/hour
Roughly 3-4 million reqs/day aggregate
Highest-traffic project on PlanetLab
*not including PlanetLab Dec 2003 upgrade
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
5
Types of Security Problems
Spammers
Bandwidth hogs
High request rates
Content thieves
Worrisome anonymity
Commonality: using CoDeeN to do things they
would not do directly
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
6
The Root of All Trouble
origin
CoDeeN
Proxy
(Malicious)
Client
No End-To-End
Authentication
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
7
Spammers
SMTP (port 25) tunnels via CONNECT
Relay via open mail server
Range: 100’s to 100,000 per day, per node
IRC channels (port 6667) via CONNECT
Captive audience, high port #
POST forms (formmail scripts)
Exploit website scripts
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
8
Bandwidth Hogs
Webcam trackers
Mass downloads of paid cam sites
Cross-Pacific traffic
Simultaneous large file downloads
Steganographers
Large files
small images
All uniform sizes
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
9
High Request Rates
Google crawlers
Dictionary crawls – baffles Googlians
Click counters
Defeat ad-supported “game”
Password crackers
Attacking random Yahoo! accounts
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
10
Content Theft
Licensed content theft
Journals and databases are expensive
Intra-domain access
Protected pages within the hosting site
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
11
Worrisome Anonymity
Request spreaders
Use CoDeeN as a DDoS platform!
TCP over HTTP
Non-HTTP Port 80
Access logging insufficient
Vulnerability testing
Low rate, triggers IDS
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
12
Approaches to Security
Desired: allow only “safe” accesses
No research in “partially open” proxies
Our approach
Rate limiting
Privilege separation
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
13
Rate Limiting
Minute
Hour
Day
3 scales capture burstiness
Exceptions
Login attempts
Vulnerability tests
Repetition, request spreading
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
14
Privilege Separation
Site B
Proxy
Remote
Client
Unprivileged
Request
Site A
Server
Site A
Proxy
Site A
Client
July 2, 2016
Privileged
Request
CoDeeN Reliability & Security
USENIX-04
15
Other Techniques
Limiting methods – GET, (HEAD)
Local users not restricted
Modifying request stream
Most promising future direction
Sanity checking on requests
Browsers, machines very different
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
16
Reasons for rejecting requests
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
17
Reliability in Context
“Real” information hard to get
Bearing on future p2p services
Non-dedicated nodes
Resource competition
Reliability more than just churn
Decentralized
No NOC, no human monitoring
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
18
Approaches to Reliability
Retry/failover
Penalty in latency
Multiple simultaneous requests
Wasting resources
Idempotency is not guaranteed
/dir/prog/query = /dir/prog?query
Active monitoring/avoidance
Failure duration, monitoring frequency
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
19
Active Monitoring
Local Monitoring
Resource availability of this node
File descriptors/sockets, system CPU time,
DNS lookup performance, uptime, load
average, free disk space
Peer Monitoring
UDP heartbeat – local monitoring data
HTTP/TCP “wget” fetch
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
20
Monitoring Implications
Missed heartbeats
Bad link, node down
Slow acknowledgements
Overloaded node
Connect failures
Resource exhaustion
Selective port filtering
Application/OS bugs
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
21
Node Avoidance Counts/Causes
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
22
Node Stability
90%
50%
20-30 min
6-7 min
8%
30 sec
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
23
DNS Problems
DNS Lookup Failure
Cacheable DNS name lookup > 5 secs
Local failures
Overloading, cron jobs, misconfigurations
Generally ~10% DNS showing problems
Critical in CoDeeN’s operation
No response from reverse proxy
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
24
Solutions to DNS Problems
Avoiding faulty nodes
Map objects in a page to same proxy
Reduce DNS lookups
Persistent connection
CoDNS
Middleware to provide reliable DNS service
Effectively removes DNS problems
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
25
Daily
Request
Volume
x 10
num of requests
6
10
9
8
7
6
5
4
3
2
1
0
06/03
July 2, 2016
rejected
requests
08/03
10/03
12/03
CoDeeN Reliability & Security
USENIX-04
02/04
04/04
26
Daily Client Population Count
16000
14000
12000
10000
8000
6000
4000
2000
0
06/03 08/03 10/03 12/03
July 2, 2016
02/04 04/04 06/04
CoDeeN Reliability & Security
USENIX-04
27
Lessons & Directions
Few substitutes for reality
Non-dedicated hardware really interesting
Failure modes not present in NS-2
Current measures pretty effective
Very slow arms race
Breathing time for better solutions
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
28
Future Work
Robot detection
Abusers are usually robots
Machine learning, high dim clustering
CoDeploy
Efficient large-file distribution service
Dynamic split/reassembly of GBs via HTTP
CoDNS
Faster, more reliable/predictable DNS service
Fully operational, used by CoDeeN
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
29
More Info
http://codeen.cs.princeton.edu
Thanks:
Intel, HP, iMimic, PlanetLab Central
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
30
Effectiveness of Monitoring
Instability patterns of nodes important
(In)stability duration
Stability measure
No status change over two intervals
Stable time pretty dynamic!
Monitoring is effective
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
31
Monitors & Other Venues
Routinely trigger open proxy alerts
Educating sysadmins, others
Really good honeypots
6000 SMTP flows/minute at CMU
Spammers do ~1M HTTP ops/day
Early problem detection
Failing PlanetLab nodes
Compromised university machines
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
32
Security Concerns
Use a popular protocol
HTTP
Emulate a popular tool/interface
Web proxy servers
Allow open access
With HTTP’s lack of accountability
Be more attractive than competition
Uptime, bandwidth, anonymity
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
33
Attempted SMTP Tunnels/Day
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
34
UDP Heartbeat Lowdown
…
…
Liveness:
MissAcks:
LateAcks:
NoFdAcks:
VersProb:
MaxLoads:
SysMxCPU:
WgetProx:
WgetTarg:
July 2, 2016
..X..
10w00
00000
00000
00000
41022
81011
00w00
11w11
X
..X..
0
00001
0
00000
0
00000
0
00000
1
11111
1
11111
1
00100
3
10301
.....
00000
00000
00000
00000
11141
11151
00010
01021
.X.XX
0w066
00000
00000
00000
20344
10656
0w110
1w220
CoDeeN Reliability & Security
USENIX-04
.....
00010
00000
00000
00000
11514
11615
00000
00111
...X.
000v0
00000
00000
00000
14204
15564
000s0
101t0
.....
00020
00000
00000
00000
11111
11111
00010
11121
35
Challenges in Deployment
Security
HTTP = Popular protocol
Open to public access
Provides a level of indirection for abusers
Reliability
Non-dedicated resources
CoDeeN depends on reverse proxy
Unreliability leads to service interruption
July 2, 2016
CoDeeN Reliability & Security
USENIX-04
36