Akamai “War” Stories
Bruce Maggs
Akamai’s First Network Connection
We moved into our offices at 201 Broadway
at midnight, December 1, 1998, and built
our first cluster that night.
An important potential investor was coming
to visit on December 2. But our Internet
service provider didn’t show up on
December 1!
We had to engineer a solution!
Lost in Space
• The most worrisome attack we ever faced:
• One of our servers was receiving properly
•
•
•
authenticated messages from an unknown host
Fortunately, the messages were not formatted
properly and were discarded
After two days of investigation, we discovered
that the “attacker” was an old Akamai server that
we had lost track of
It had been sending these messages for months!
4
David is a Night Owl
• Your servers aren’t responding!
• Why don’t you support half-closed
connections?
• Why don’t you support “transactional”
TCP?
• (Why would transactional TCP be bad
for Akamai?)
5
The Dreaded Double Header
• http://images.xyz.com/logo.gif
- customer has delegated images.xyz.com to
Akamai, registered image server
• http://images.xyz.com/images.xyz.com/logo.gif
- didn’t work for Dave, but worked for me!
• Akamai server strips off first header, sends
GET /images.xyz.com/logo.gif to customer image
server
• 5 of 8 customer image servers had been
patched to ignore /images.xyz.com
6
Steve can’t see the new Powerbook
• Steve’s assistant Eddie explains the
problem
• I spend all night poring through the logs
• Eddie sneaks into Steve’s office
• Mystery solved
7
Packet of Death
• Akamai servers take care of each other
• A router in Malaysia is taking down our
whole system!
• The mysterious 570-byte MTU
• The “final” Linux kernel isn’t so final
• 2.0.36 (Nov. 1998)  2.0.37 (June 1999)
8
The “Magg Syndrome”
• We “hijack” a customer’s site?
• I become the most hated person on the
Internet
• We isolate the problem (nine months of
work)
• Nobody cares?
9
Don’t do this at home
•
•
•
•
•
Irate end user threatens to go to police
Akamai is attacking my home system!
It’s in the logs.
It all began in a Yahoo! chat room
Have your lawyers call our lawyers
10
BIND Miseries
•
•
•
•
Open-source DNS server code
Messy, buggy implementations
Our customers still run old versions!
BIND 4.8 TTL issue
• Refresh attempt when 15 minutes left
• Success if new list of IP’s overlaps with old
•
list of IP’s
Otherwise, refuse to resolve for next 15
minutes!
11