BARE METAL SOFTWARE DEFINED NETWORKS, AND OPEN SOURCE Rob Sherwood Big Switch Networks, CTO BANV, May 2014 OUTLINE •
•
•
•
Mo6va6on: Open Networking and Hyperscale Modern SDN Big Switch Technologies Open Source SoKware •
•
•
•
Open Network Linux OF-­‐DPA Indigo OpenFlow agent Loxi + Floodlight •  Conclusions and Community © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 2
MOTIVATION Do you
?
THE DC NETWORKING REVOLUTION What are they doing? 1) SDN soVware 2) Bare metal switch hardware © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 5
ANDROMEDA © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 6
WHAT DOES BIG SWITCH DO? Help you get from here to there We build SDN SoVware... …that runs on Bare Metal Switch Hardware (to reduce complexity) © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M (to reduce cost) …to monitor networks and build cloud fabrics (to deliver Bare Metal SDN) 7
SDN ARCHITECTURAL EVOLUTION BEFORE Automa[on Tool APIs SDN App North-­‐bound APIs SDN Controller OpenFlow CP/DP Separa[on (Thick) N etOS OF Network HW •  Too many moving parts (mul[ple SDN SW vendors) •  OF agent SW from HW vendor! •  Limited access to switch ASIC access © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 8
SDN ARCHITECTURAL EVOLUTION BEFORE NOW Automa[on Tool Automa[on Tool APIs North-­‐bound APIs SDN App North-­‐bound APIs SDN Controller OpenFlow SDN App SDN Controller CP/DP Separa[on OpenFlow & Extensions (Thick) N etOS OF Network HW •  Too many moving parts (mul[ple SDN SW vendors) •  OF agent SW from HW vendor! •  Limited access to switch ASIC access INC. ROPRIETARY ONFIDENTIAL ©©2014 2 0 1 4B IG B I SGWITCH S W INTETWORKS, C H N E T W
O R KWSWW.BIGSWITCH.COM , I N C . W W W P. B
I G S W I T CAHND . CCO
M Openflow is just API (Thin) S DN OS Network HW •
•
•
•
SDN SW solu[on from single vendor Exactly like the hypervisor/server model Full access to switch ASIC Accelerate HW/SW disaggrega[on 9
SDN ARCHITECTURAL EVOLUTION Cloud Big Tap Fabric NOW Automa[on Tool North-­‐bound APIs Big Switch Network’s Commercial Products SDN App SDN Controller Open Source (Thin) S DN OS Network HW OpenFlow & Extensions •
•
•
•
INC. ROPRIETARY ONFIDENTIAL ©©2014 2 0 1 4B IG B I SGWITCH S W INTETWORKS, C H N E T W
O R KWSWW.BIGSWITCH.COM , I N C . W W W P. B
I G S W I T CAHND . CCO
M OpenFlow is just an API SDN SW solu[on from single vendor Exactly like the hypervisor/server model Full access to switch ASIC Accelerate HW/SW disaggrega[on 10
ISN’T NETWORKING ALREADY OPEN? Faster Throughput = More commercial value = More Proprietary All Hardware Specs Public X86 Servers ß Open Binary SDK for Packet Forwarding NPU, FPGA, SoC Closed OS, Open Control Plane APIs DC – 1U ToRs, Spines Closed OS, Open APIs for Policy, Stats Mul[-­‐chassis Routers 10 Gb/s 100 Gb/s 1 Tb/s 10 Tb/s Aggregate Throughput à © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 11
ISN’T NETWORKING ALREADY OPEN? Faster Throughput = More commercial value = More Proprietary All Hardware Specs Public X86 Servers ß Open Binary SDK for Packet Forwarding NPU, FPGA, SoC Closed OS, Open Control Plane APIs DC – 1U ToRs, Spines This talk DC – 1U ToRs, Spines Closed OS, Open APIs for Policy, Stats Mul[-­‐chassis Routers 10 Gb/s 100 Gb/s 1 Tb/s 10 Tb/s Aggregate Throughput à © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 12
MODERN SDN SDN BASICS Manage Your Network Like a “Big Switch” 2 3 4 5 6 1
2 3 4 5 6 1
© 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 14
SDN BASICS Supervisor 1 Supervisor 2 Line Card Line Card Line Card Line Card Fabric Backplane Manage Your Network Like a “Big Switch” 2 3 4 5 6 1
2 3 4 5 6 1
Line Card Chassis-­‐based Fabric © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 15
Line Card Line Card Line Card Chassis-­‐based Fabric © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M Leaf Leaf Leaf Line Card 2 3 4 5 6 1
Leaf Line Card 2 3 4 5 6 1
Spine Supervisor 2 Controller 2 Spine Supervisor 1 Fabric Backplane Manage Your Network Like a “Big Switch” Controller 1 Spine SDN BASICS Leaf-­‐Spine Fabric 16
Line Card Line Card Line Card Chassis-­‐based Fabric These designs are fundamentally the same! © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M Leaf Leaf Leaf Line Card 2 3 4 5 6 1
Leaf Line Card 2 3 4 5 6 1
Spine Supervisor 2 Controller 2 Spine Supervisor 1 Fabric Backplane Manage Your Network Like a “Big Switch” Controller 1 Spine SDN BASICS Leaf-­‐Spine Fabric 17
CLOSEDFLOW VS. OPENFLOW WWCFD? Same Packet Forwarding Architecture. •  For the rest of the talk, before you ask a ques6on: •  “WWCFD – What Would ClosedFlow do?” •  Answer: “OpenFlow/SDN probably does the same thing” •  Is this really the right ques6on? •  SDN is a design paradigm, not a solu[on onto itself -  Unless you’ve got an army of programmers… •  Customer benefits are automa[on and simplicity, not “SDN” •  Just like OO Programming, DevOps, etc. © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 18
DECOUPLE CONTROL FROM FORWARDING OpenFlow Controller OFDatapat
OFDatapath h OFDatapath OFDatapath OFDatapat
OFDatapath
h •  Reduce number of management touchpoints •  Mapping from datapaths to controllers a crucial network design ques[on OpenFlow does not imply single point of failure! © 2 0 1 3 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 19
DECOUPLE CONTROL FROM FORWARDING OpenFlow Controller OFDatapat
OFDatapath h OFDatapath OFDatapath OFDatapat
OFDatapath
h OpenFlow Controller OFDatapat
OFDatapat
h OFDatapath h OFDatapat
OFDatapat
h OFDatapath
h OpenFlow does not imply single point of failure! Allows load balancing © 2 0 1 3 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 20
DECOUPLE CONTROL FROM FORWARDING OpenFlow Controller OFDatapat
OFDatapath h OFDatapath OFDatapath OFDatapat
OFDatapath
h OpenFlow Controller OFDatapat
OFDatapat
h OFDatapath h OFDatapat
OFDatapat
h OFDatapath
h OpenFlow does not imply single point of failure! Allows load balancing and failover © 2 0 1 3 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 21
PACKET FORWARDING ABSTRACTION: FLOW TABLE Sequence of tables in a packet processing pipeline Priority Match Ac6on List 500 IP.proto=6 TCP.dst=22 TTL-­‐-­‐, Fwd:port 3 200 IP.dst= 128.8/16 * Queue: 4 Flow TTable Flow Flow Table able 100 DROP •  Exis[ng networking hardware actually very flexible • Trade-­‐off: Large + narrow versus small + wide match tables •  Ac[ve work in the Open Networking Founda[on to bring OpenFlow to feature parity with “closed flow” © 2 0 1 3 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 22
SWITCH SCALING: UNDERSTANDING TABLES Classical OpenFlow versus Modern: Proac[ve and Mul[-­‐table OpenFlow Processing Pipeline L2 Table: L3 Table: ~100K+ ~100K+ Entries Entries VLAN Table Early OpenFlow implementa[ons only used the most flexible table. •  Didn’t scale well •  Scale forced reac[ve popula[on •  Reac[ve caused controller load and addi[on scale problems ACL Table: ~2k Entries Our Modern OpenFlow implementa[on leverages all tables •  Scales like tradi[onal •  Allows proac[ve popula[on •  Proac[ve reduces controller load and allows for headless control Egress Table OpenFlow does NOT imply Flow-­‐based Networking © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 23
BIG SWITCH TECHNOLOGIES WHAT’S INSIDE A SWITCH? Applica[on Network OS Hardware Driver Box Silicon © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 25
COMPONENT ECOSYSTEM AND BARE METAL Applica[on Network OS Driver Single Vendor Closed Product Box Single Vendor Closed Product ODM Box SDN Controller Vendor OpenFlow SDN Hardware Vendor ODM Chip Silicon Tradi[onal Networking (past) Tradi[onal Networking (today) © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M Tradi[onal Network Stack/OS Vendor ODM Box ODM Chip OpenFlow Model Bare Metal Vision 26
SWITCH LIGHT IS SDN ON BARE METAL •  The OpenFlow Ecosystem Model didn’t work for us •  Hard: one OF agent that is all things to all people •  OF Wire protocol is (mostly) fine, but needs work for exis[ng hardware •  Economic incen[ves are backwards for SDN hardware vendors •  Out-­‐of-­‐phase release cycles caused low feature velocity •  Switch Light Model: download OS image and OF agent from Controller •  Easy : Customize switch soVware to applica[on and controller version •  Add extensions to wire protocol to validate before standardiza[on •  Perfect economic incen[ves •  One engineering team à high feature velocity © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 27
BARE METAL + MERCHANT SI: LOWERS FABRIC COSTS Financial Benefit of SDN Business Agility is Increased Revenue Reduced OpEx Reduced CapEx CapEx Savings is Not Trivial •  Brand name network vendors are using the same components •  “Bare metal” is more than “white box” •  Name brand vs. no-­‐name vendors •  Buy your network hardware just like servers •  Vendors already successful in the server model have advantage © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 28
ARCHITECTURE IS OPEN FROM THE DATA PLANE UP •  Open data plane •  Re-­‐use exis[ng virtualiza[on formats: no proprietary protocols/new hardware •  Compa[ble: deployed firewalls, IDS, WAN accelerators, Server offload NICs •  Open control plane •  OF1.3: All extensions are open source and will be taken to ONF •  Open REST APIs is primary interface: CLI is just a REST client •  Integra[on with OpenStack/Quantum and custom BSS/OSS systems •  Open source components: soKware engineering best prac6ce •  Accelerates adop[on, eases por[ng •  Open Network Linux: bare metal Linux distribu[on in OCP •  Indigo Open Flow agent: many 3rd par[es have ported to their hardware © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 29
Leaf Server Server Leaf Server Server Leaf Server Server Leaf Spine Spine Spine FABRIC USE-­‐CASE (1/3): CONNECTIVITY, POLICY Router Router © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M •  Most common fabric use-­‐case •  Policy: connec[vity, ACLs, QoS •  Op[onal VM orchestra[on •  e.g., OpenStack, HyperV 30
FABRIC USE-­‐CASE (2/3): MONITORING Leaf Leaf Exis[ng Produc[on Network Leaf Leaf Spine Spine Spine Tap Traffic IDS VoIP QoS Tool © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M •  BigTap: Scale-­‐out monitoring •  Monitoring “as a Service” •  Programmable via REST •  RBAC •  Overlapping policy support •  Physically isolated from prod •  In/out heavily over-­‐subscribed •  Load balance across tools •  “My first SDN App” 31
Leaf DUT DUT Leaf DUT DUT Leaf DUT Leaf Spine Spine Spine FABRIC USE-­‐CASE (3/3): PATCH PANEL Packet Generator DUT Packet Generator *DUT: Device Under Test © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M •  Programma[cally create large virtual L1 topologies •  Automate complex tests without rewiring •  Blindly forward packets by port •  Similar to Pseudo-­‐wire •  Even CRC failures, runts, etc. •  Non-­‐product: in our Q/A labs 32
OPEN SOURCE SOFTWARE SWITCH LIGHT ARCHITECTURE Big Network Controllers Legend Open Network Linux BSN Open BSN Closed 3rd Party Closed Source CLI Switch Light OS ZTN Loader SSH Fan Control NTP Syslog SNMP LibC on Debian Wheezy Base Distribu[on ONL Linux Kernel I2C GPIO Device Trees OpenFlow Agent Loxi Indigo Indigo/ASIC Driver ASIC SDK ASIC Switch Light is our Indigo OpenFlow Agent running on Open Network Linux on x86 or ASIC-­‐based hardware. © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 34
OPEN NETWORK LINUX A Linux distribu[on for bare metal switches •  Open-­‐sourced commercial code, build scripts, drivers from Switch Light •  Part of Open Compute Project (OCP) •  Project goals: •  Improve percep[on/de-­‐risk bare metal •  “Crowd source” larger Switch Light HCL •  Released mid-­‐January: Ini6al results •  Strong community response from ODMs, OCP •  Accton self-­‐suppor[ng three new boxes, more ODMs interested •  Other vendors inves[ga[ng ONL for their own purposes •  Info: h{p://opennetlinux.org – under construc[on © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 35
ONL – SUPPORTED HARDWARE X86 versus PPC Discussion •  ONL Support is Pre-­‐cursor to Switch Light Support •  Working on OCP box from Interface Masters with x86 support © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 36
ONL IS MULTI-­‐PLATFORM Support many boxes from the same code-­‐base Interface Master’s
Open
Network
Linux:
•
•
•
•
•
Kernel
Drivers
Loader
Work flow
Build
scripts
•  Manage.
Model
X86 Arch
x86 VM
others?
Quanta LB9, LY2, LY5
PPC
Accton 5652
Delta, Alpha, etc.
ARM?
© 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M ???
37
INSTALL USING ONIE THEN BOOT ONL Boot Logic:
~64MB
uBoot
ENVs
ONIE
Free
Space
Boot
Flash
~2GB
ONL
1.  uBoot POSTs
Loader
2.  $nos_bootcmd is read from ENVs
ONL
config
3.  run $nos_bootcmd
•  If $nos_bootcmd returns, run ONIE
SWI’s:
•  On install, ONIE sets $nos_bootcmd ONL
to load ONL loader
SWI #1
4.  Loader downloads specified SWI URL (cached)
if not cached
5.  Loader mounts rootfs as ramdisk with
overlayfs
Mass
6.  ONL loader kexec’s SWI kernel
Storage
© 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 38
PERSPECTIVE RELATIVE TO ONIE ONL is a Network Opera[ng System (NOS) for ONIE ~160 MB ~3MB ~16MB Main Network OS Image (.swi) (w/real binaries) ONIE Normal Full-­‐featured First boot Loader Boot Loader (w/BusyBox) Open Network Linux Github.com/ onie/onie Common kernel and DTS files?? © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 39
© 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 40
OpenFlow Datapath Abtrac[on: OF-­‐DPA Indigo driver open OF-­‐DPA SDK Chip closed OF-­‐DPA API © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 41
LOXI IS FULLY OF1.3.1 git://github.com/floodlight/loxigen Single OF Wire Desc LOXI-­‐
GEN C Backend libLOCI.a Indigo Java Backend OpenFlow
J-­‐LOXI Floodlight Python Backend Pylib openflow OFTest Wireshark Backend © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M Wireshark Plugin (Lua) Wireshark 42
FLOODLIGHT SDN CONTROLLER © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 43
CONCLUSION •  Big Switch builds SDN solu6ons on bare metal hardware •  Commercial Products: BigTap, Cloud Fabric •  SDN has evolved significantly •  Mul[-­‐table, proac[ve, bare metal •  Lots of Open Source projects •  Take the pieces or the whole: ONL •  Join at: opennetlinux.org/community •  Comments and feedback welcome © 2 0 1 4 B I G S W I T C H N E T W O R K S , I N C . W W W . B I G S W I T C H . C O M 44