Managing Open vSwitch Across
a Large Heterogeneous Fleet
Chad Norgan
Systems Engineer
BeardyMcBeards in #openvswitch
About Rackspace
Over $1B Annualized Revenue
300,000+
Customers
≅70 PB Stored
We Serve
FORTUNE®
60% 100
OVER
Global Footprint
Customers in
120+ Countries
5,000+
OF
THE
Rackers
9 Worldwide
Data Centers
Portfolio of
Hosted Solutions
Dedicated - Cloud - Hybrid
2
www.rackspace.com
Rackspace’s Public Cloud
Large Fleet
• Tens of thousands of hypervisors
• Hundreds of thousands of
Heterogenous
•
•
virtual machines
• 1 - 3+ Interfaces
•
•
•
•
Several different hardware
manufacturers
Several XenServer major versions
(sometimes on varying kernels)
Five networking configurations
Six production public clouds
Six internal private clouds
Various non-production environments
Networks Available to Customers
Our History With OVS
•
Rackspace has used Open vSwitch since the 0.9 version
•
Behind most of First Generation Cloud Servers (Slicehost)
•
Powers 100% of Next Generation Cloud Servers
•
Upgraded OVS nine times since the launch of Next Gen
Public Cloud in August 2012
Why We Use OVS
•Service provider features:
–Overlay Networks
–QoS
–VLAN Tagging
–Port Security
–LACP
•Software = Flexible
•Upgrades are easier than hardware
Our Favorite Improvements
OVS 1.7 • Save & restore datapath flows during kmod reload
OVS 1.9 • Logging removed from main loop, faster flow setups
OVS 1.10 • Collapsed data path & flow-eviction-threshold raised to 2500
OVS 1.11 • Megaflows & wildcarding
OVS 2.0 • Multi-treading!
OVS 2.1 • flow-limit replaces flow-eviction-threshold & TCP flags
Example: Busy HV With Syslog Collector
Mission Accomplished!
We moved the bottleneck!
New bottlenecks:
● Guest OS kernel configuration
● Xen Netback/Netfront Driver
Challenges of Upgrading OVS
•Matching the OVS kernel module to both the running and
staged kernel
•Hypervisor updates often come with a newer kernel
•We often don’t immediately reboot
•Running kernel != Kernel at next reboot
•Detect both kernels and install both sets of OVS kernel
modules
•Heterogeneous Scale
OVS Upgrade Solution
• Playbook-style upgrades
• Asynchronous plays with
parallel limits
• Extensible
• Easy to build validations and
pre-checks to prevent
unwanted impact
Architectural Basics
VIF
Tunnel Encapsulation
VIF
Integration
Bridge
Patch
Port
VIF
PIF
Interface
Bridge
PIF
Ansible + OVS = Flexible Network Rewiring
VIF
Interface
Bridge
VIF
VIF
Integration
Bridge
PIF
PIF
Ansible + OVS = Flexible Network Rewiring
VIF
Public Net
Bridge
Interface
Bridge
VIF
Integration
Bridge
VIF
PIF
PIF
Ansible + OVS = Flexible Network Rewiring
VIF
Public Net
Bridge
Interface
Bridge
VIF
Integration
Bridge
VIF
PIF
PIF
Ansible + OVS = Flexible Network Rewiring
VIF
VIF
VIF
Public Net
Bridge
Interface
Bridge
PIF
Service Net
Bridge
Integration
Bridge
Tunnel
Encap
Ansible + OVS = Flexible Network Rewiring
VIF
Public Net
Bridge
VIF
Service Net
Bridge
VIF
Cloud Net
Bridge
Interface
Bridge
Patch Port
Integration
Bridge
PIF
Tunnel
Encap
Ansible + OVS = Flexible Network Rewiring
Public Net
Bridge
VIF
Public Net
Bridge
Bridge_old
Patch Port
Interface
Bridge
PIF
Measuring OVS – PavlOVS.py
Publishes metrics to StatsD/Graphite
• Per bridge byte, packet, open flow count
• Datapath hit, missed, lost, flow counts
• Open vSwitch CPU utilization
• Instance count
• Tunnels configured and in fault state
2000 Eviction Threshold
Hit, Miss, Lost
OVS CPU
The OVS Of Our Dreams
• Connection Tracking
• More (efficient) performance
• JSON Output from ovs-*ctl commands
THANK YOU
RACKSPACE®
|
1 FANATICAL PLACE, CITY OF WINDCREST
US SALES: 1-800-961-2888
© RACKSPACE LTD.
|
|
US SUPPORT: 1-800-961-4454
|
|
SAN ANTONIO, TX 78218
WWW.RACKSPACE.COM
RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES.
|
WWW.RACKSPACE.COM
THANK YOU
QUESTIONS?
RACKSPACE®
|
1 FANATICAL PLACE, CITY OF WINDCREST
US SALES: 1-800-961-2888
© RACKSPACE LTD.
|
|
US SUPPORT: 1-800-961-4454
|
|
SAN ANTONIO, TX 78218
WWW.RACKSPACE.COM
RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES.
|
WWW.RACKSPACE.COM