The Netflix Open Connect Network
DKNOG – 21 March 2013
1
Today we are in 22 locations across the world...
(16 Primary, 6 “network extensions”)
2
3
How do we decide what locations to go to?
 Where is Netflix presently providing service?
 Our service is limited by the content licenses that we can buy
 For example, we are in the Nordic countries, the UK, and Ireland but
not France/Germany
 Can we interconnect with networks that are relevant?
 Going to a peering point that’s all content providers does no good
 Can we reduce the cost and improve traffic
management for networks who need to deliver
Netflix traffic to their end users?
4
Netflix in the Nordics
 We provide our streaming service to
 Denmark
 Finland
 Norway
 Sweden
 Iceland and Greenland
 Currently co-located at Telecity Stockholm 1 & 2
 Stockholm 1 is a network extension from Stockholm 2 due to
space/power constraints
 Likely to expand to a second market in the Nordics this
year
5
IXes
 Netflix delivers their Nordic traffic from four different IXes
 NetNod
 NIX
 AMS-IX
 LINX
 The latter two may not seem relevant here, but many
networks have significant capacity built there
6
Standard build - European
 A European Netflix Open Connect IX build has:
 20 Netflix Open Connect Appliances
 Two Juniper MX480 routers
 OpenGear console server and Juniper EX2200’s for OOB access
 Prewired MTP to LC cabling from Elpeus
 A “network extension” Open Connect build has:
 Two Juniper EX4550 switches
 Two MRV 40 channel passive muxes
 OpenGear console server for OOB access
7
Stockholm
8
Network Extension POP
9
Standard Build - US
 A US Netflix Open Connect IX build has:
 40 Netflix Open Connect Appliances
 60 Netflix Open Connect Flash Appliances
 Two Juniper MX960 routers
 OpenGear console server and Juniper EX2200’s for OOB access
 Prewired MTP to LC cabling from Elpeus
 A “network extension” Open Connect build has:
 Two Juniper EX4550 switches
 Two MRV 40 channel passive muxes
 OpenGear console server for OOB access
10
11
Converting some POPs to Cisco ASR
 Due to concerns with Juniper code delivery and the need
for higher density platforms, we decided in early 2013 to
integrate some Cisco ASR 9K into our network to replace
Juniper MX
 In order to deliver our traffic goals (~1T per router), we
need denser cards than the currently shipping Juniper 16
port MPC3’s
12
High Density…
 Juniper MX960 w/ MPC3’s can deliver 176 ports per
chassis
 A single fabric failure brings the chassis to 3:2 oversubscription
 Juniper MX960 w/ MPC4’s can deliver 336 ports per
chassis (oversubscribed)
 A single fabric failure (out of 3) brings the chassis to 2:1
oversubscription – unacceptable
 Cisco ASR9010 w/ Typhoon-based 36 port cards can
deliver 288 ports per chassis
 A single fabric failure brings the chassis to 3:2 oversubscription
13
High Density, continued…
 We conservatively need 130 host ports per router
 Those 130 ports can deliver 1.13T of traffic
 Assuming we deliver 1 bit outbound for every bit inbound
from a host, that puts us at 1.1:1 oversubscribed at peak
 3:2 and 2:1 just do not work for us
 Juniper MPC4’s also require brand new 12.3 code
 What version of code are you running?
14
Cabling a new ASR…
15
The Netflix Open Connect Appliance (OCA)
 Developed in response to ISP requests to help scale
Netflix traffic efficiently
 Reduces ISP cost by serving Netflix traffic from the local
ISP datacenter, CO or headend, rather than upstream
network interconnects
 Speeds up internet access for consumers to all thirdparty internet sites, because Netflix traffic is no longer a
source of middle-mile or backbone congestion
 Netflix bears the capital and maintenance costs, not ISP
 ISP provides space, power and a network port
 An OCA is a component of the Netflix CDN (vs a cache)
16
OCA Hardware
• Space optimized: 4U
high-density storage
• Power optimized for low
power/cooling
requirements (≅500W)
• Redundant 10GE optical
network interfaces
• Redundant power
supplies (AC or DC)
17
Why Deploy a Netflix OCA?
 Netflix data is a significant percentage of ISP traffic in the
markets we serve
 Serve >80% of Netflix traffic from the local ISP datacenter
 Remaining traffic can be served by upstream OCA’s, peering
 Saves on transit, transport and other upstream scaling costs
 Provided free of charge to participating ISPs
 ISPs with > 3 Gbps of Netflix traffic
 ISP provides rack space, power, 10 Gbps optical port(s)
OCA Operation
 Used exclusively for Netflix content
 Completely integrated with the Netflix content delivery system
 Greater effectiveness than transparent or proxy caches (>80% with
Netflix OCA, versus 20-50% with other caches)
 Efficient content fill mechanisms
 Outside peak times (ISP selected time and BW per OCA)
 OCA offline during fill, staggered fill recommended.
 Fill can source from neighboring, peer or transit OCA’s
 ~5TB fill per day (i.e ~4Gb/s for ~2.5 hrs)
 OCA must be reachable by end users (port 80)
Directing Clients to OCAs
Netflix Control Servers
Broadband ISP
Netflix OCA
3. Client connects to local OCA
4. Local OCA delivers video stream
• User routing is done by
Netflix control servers,
not dependent on client
DNS configuration
• Request is routed to the
nearest available OCA
• Working sets of popular
content deliver up to
>80% network offload
Network Positioning
 OCA’s are installed close to aggregations of end users
 Netflix works with ISPs to determine proper location
and user-to-OCA mapping
 ISP Controls Traffic Flow
 End-user netblocks are associated with OCA via BGP advertisement
 Netflix honors AS path-length and MEDs allowing ISP to configure OCA
hierarchy and failover
 OCA’s only serve content to netblocks provided by ISP
 ISPs choose peak throughput each OCA is allowed to generate
The Netflix ISP Speed Index
 Published monthly
 Helps consumers make an intelligent choice about their
broadband provider
 We want our users to have the best experience possible
 We also want to help networks deliver Netflix traffic as
efficiently as possible
22
23
A recently enabled network (night of 18/03/13)
No marked increase in bandwidth utilization, but a more stable experience for users
24
Questions?
Additional information:
http://openconnect.netflix.com