MARRIAGE OF OPENSTACK
WITH KVM AND ESX AT
PAYPAL
MULTI-VENDOR AGILITY
Open Stack Summit – Hong Kong - 2013
ABOUT PAYPAL
PayPal offers flexible and innovative payment solutions for consumers
and merchants of all sizes.
• 137,000,000 Users.
• $300,000 Payments processed by PayPal each minute.
• 193 markets / 26 currencies.
• PayPal is the World’s Most Widely Used Digital Wallet.
2
WHY WE VIRTUALIZED ON ESX
• In 2011/2012, 90% of the PayPal front-end was virtualized on ESX
5.0u1
• Primary Criteria
− Stability, performance, industry expertise, availability of experts
• Standardized on VCE VBLOCK© for initial implementation
• Fully consumable API
• Load-test harness well understood in industry (specInt -> vMark)
− Predictable scaling pattern for horizontally scaled workloads
3
CLOUD
4
PAYPAL INTERNAL CLOUD
2012/2013 Shift toward an internal cloud model
• Shift from Enterprise design model to cloud-based design
• Elastically scale and self-heal infrastructure to accommodate
unpredictable usage patterns of customers and internet commerce
• Separate rapidly iterating customer experiences from core
services
• reduce overall cost per transaction within the environment
5
CLOUD IS THE GREAT ENABLER
ENABLE THE DEVELOPER
ENABLE THE BUSINESS
One-Click
Developer
Self Service
Global
Compute &
Data
Fulfillment
Payment Delivery
SelfOrganizing &
Optimizing
Infrastructure
System Intelligence
Driven Operation
Code
6
Deploy
Enjoy
PAYPAL CLOUD PLATFORM –
GUIDING PRINCIPLES
• Technology
− Adopt Open Source Solutions where ever possible
− No Vendor Lock-in
− Industry Best Practices
− Leverage Industry/ebay Inc Investments
• Functionality
− Self-Service tool for application life cycle management.
− Robust Automation & Orchestration
− Seamless On-Demand Capacity Fulfillment
7
OPENSTACK
PayPal deploying Openstack in order to help transform our global
infrastructure into an agile and open cloud platform.
Agility - time to market for customer facing services
Agility - speed to service developer requests for VM resources
Agility – utilize the engineering culture of PayPal to
deliver specialized cloud services where needed
8
TECHNOLOGY STACK
User
Interface
Operations Portal
DEVS Deployment Portal
Horizon, Ceilometer
Traffic Mgmt
Monitoring
Metering
Stages
Workflow
Monitoring
Orchestration Engine
Orchestration
Cloud Formation (Heat)
Foundational
Services
Nova, Cinder, Swift, Keystone, Quantum, Horizon
Software
Infrastructure
Cobbler
ISC DHCP
Hardware
Infrastructure
x86 Compute
Salt
BIND
Local Storage
RHEL 6.x
Network
LBaaS, DNSaaS
FWaaS
Hypervisor
Zabbix
Load
Balancer
PP Specific
9
CLOUD BEFORE INTEGRATION
WEB
F
Z
F
Z
F
Z
KVM
Local Disk
“Stateless & Disposable”
F
Z
MID
Cloud Management Zone
VCenter Management
F
Z
F
Z
F
Z
KVM
Local Disk
F
Z
F
Z
ESX
5.0u2
Shared
Storage
F
Z
ESX
5.0u2
Shared
Storage
F
Z
ESX
5.0u2
Shared
Storage
Physical
Non-virtualized
F
Z
ESX
5.0u2
Shared
Storage
Physical
Non-virtualized
DATABASE & RESTRICTED ZONE
FZ = Logical Fault Zones
SIDE-BY-SIDE
11
CLOUD AFTER INTEGRATION
F
Z
F
Z
F
Z
F
Z
F
Z
WEB
F
Z
KVM
Local Disk
ESX 5.0u2
Shared Storage
Physical
Non-virtualized
MID
Cloud Management Zone
F
Z
KVM
Local Disk
ESX 5.0u2
Shared Storage
Physical
Non-virtualized
DATABASE & RESTRICTED ZONE
COMPARING
But isn’t Openstack a direct replacement for ESX? Why would
you keep them both?
ESX/Vsphere != Openstack
NOVA != vSphere || vCenter || ESXi
NOVA =~ vCD, vCAC
KVM =~ ESX
To connect to any hypervisor, the Openstack cloud ‘proxies’
connections to any supported hypervisor via Nova. That
abstracts the ‘Cloud’ from the hypervisor
13
BRINGING ESX ‘INTO’ THE CLOUD
• Equivalent functionality on KVM and ESX
• Full birth to death lifecycle management of virtual machines
− Build new, power on, power off, console, rebuild, delete
• Auto-configuration of host resources following t-shirt sizes
standards
− CPU, RAM, NIC, IP, OS Version
• IP Address Management
• Build from “Snapshot”/”Template”
• Deploy resources following appropriate fault zone model
• Must work from within single Horizon/Asgard interface
14
HYPERVISOR REQUIREMENTS
• ESX 5.1
− 5.0 works but too many back-ports (for us) / tweaks
• Single security zone per hypervisor
− No sharing of confidential & non-confidential on same hardware (PCI)
• Openstack management network communication
− This is NOT necessarily the VKERNEL network
15
STORAGE REQUIREMENTS
• “Shared storage” required
− Data Store Cluster
− Single Data Store support
• DRS Enabled with auto-placement
• Data Stores must be created in advance
− No Cinder support
16
OPENSTACK GRIZZLY
⁃ OpenSt ack Command Line Tools (nova-client, swif t-client, et c.)
⁃ Cloud M anagement Tools (Right scale, Enst rat ius, et c.)
⁃ GUI t ools (Cyberduck, iPhone client, et c.)
Int er net
OpenSt ack
Object API
OpenSt ack
Comput e API
OpenSt ack
Image API
OpenStack
Identity
API
OpenStack
Dashboard
HTTP(S)
Amazon
Web Ser vices
EC2 API
VNC/ VMRC
/ Spice
OpenSt ack
Block St orage API
Hor izon
OpenSt ack
Net wor k API
OpenStack
Object API
OpenStack
Image API
OpenSt ack
Object API
swif t-proxy
OpenStack Compute
API /
Admin API
OpenStack
Identity
API
OpenSt ack
OpenSt ack
Block St orage API Block St orage API
nova-api
OpenSt ack
Image API
glance-api
(OS, EC2, Met adat a, Admin)
nova-comput e
nova-cert/
objectstore
glance-regist r y
cont ainer
object
cinder-api
nova-console
nova-*proxy
OpenSt ack
Image
API
memcached
account
OpenSt ack
Net wor k API
OpenSt ack
Net wor k API
HTTP(S)
cinder-volume
nova
dat abase
object
st ore
OpenStack
Identity
API
OpenSt ack Object St ore
quant um
agent (s)
Queue
net wor k
provider
quant um
dat abase
quant um
plugin(s)
Queue
volume provider
Queue
hyper visor
cont ainer
DB
cinder-backup
libvirt, XenAPI, et c.
glance
dat abase
account
DB
quant um-ser ver
cinder
dat abase
nova-conduct or
nova-consoleauth
cinder-scheduler
http://www.solinea.com
OpenStack
Identity
API
OpenSt ack
Image Ser vice
nova-scheduler
OpenSt ack Comput e
OpenSt ack
Block St orage
OpenSt ack
Net wor k Ser vice
OpenStack
Identity
API
OpenStack
Identity API
keyst one
(ser vice & admin APIs)
OpenStack
Identity
Service
t oken backend
cat alog
backend
policy
backend
OpenStack Object API
OpenStack
Identity
API
OpenStack
Identity
API
ident it y
backend
ITS ALL ABOUT NOVA
CONFIG OF NOVA
Nova is the project name for OpenStack Compute, a cloud computing fabric controller,
the main part of an IaaS system. Individuals and organizations can use Nova to host
and manage their own cloud computing systems.
#compute_driver = libvirt.LibvirtDriver
compute_driver = vmwareapi.VMwareVCDriver
Can be multiple
vmwareapi_host_ip=192.168.20.50
clusters now!
vmwareapi_host_username=root
vmwareapi_host_password=vmware
vmwareapi_cluster_name=openstack_test
vmwareapi_wsdl_loc=https://192.168.20.50/sdk/vimService.wsdl
Vcenter 5.1 Appliance
19
Confidential and Proprietary
GLANCE AND IMAGES
Rules for Glances images for VMWare
• Saved in VMDK Format
• Imported as VMDK Format
• Thick Provisioned VMDK Required
• No split VMDK allowed (must be merged)
• In a multi-hypervisor cloud, all images are separate
‘per hypervisor’ (no launching KVM VM’s on ESX)
glance add name=”MYMACHINE.vmdk" disk_format=vmdk container_format=bare
is_public=true vmware_adaptertype="lsiLogic" vmware_disktype="preallocated"
vmware_ostype="otherGuest" < /path/to/MYMACHINE.vmdk
20
Confidential and Proprietary
BUILDING AND INSTALLING OS
• Kickstart
• Build a small root disk
• Use kickstart to image machine
• Post-install with puppet to customize machine and
add additional mount points depending on
application requirements
• Image Deploy
• Currently does not support ‘config-drive’
• Need Guest Tools to ‘duplicate’ functionality
21
Confidential and Proprietary
WHAT ABOUT THE NETWORK
•
22
Quantum requires NVP 3.2
• Cannot talk directly to VSphere API to allocate VDS
Port to NIC
• Uses vAPP – integration bridge or native in 5.5
• Configured as separate transport zone within Nicira
Confidential and Proprietary
WHAT’S LEFT
• Component “at-scale” testing
• Currently manage “tens” at a time, need to
move to “hundreds” or “thousands”
• Most fixes in Havanna, every bug-fix needs to be
reviewed and possible back-ported to Grizzly
• Multiple Data Store enumeration on a cluster
• Full Certification on VCE VBLOCK with Vision
Intelligent Operations, auto-upgrades, and full
Openstack support of all components
23
Confidential and Proprietary
READING MATERIALS
•
•
•
•
•
•
http://www.solinea.com/2013/06/15/openstack-grizzly-architecture-revisited/ - Ken Pepple
http://www.slideshare.net/kenhui65/getting-started-with-openstack?ref=http://cloudarchitectmusings.com/2013/06/16/getting-started-with-openstack/ - Kenneth Hui
http://docs.openstack.org/trunk/openstack-compute/admin/content/config-drive.html - config-drive doc
http://docs.openstack.org/trunk/openstack-compute/admin/content/vmware.html - Openstack VMWARE doc
http://www.ebay.com - Buy It Now
http://www.paypal.com - and then Pay for it Here!
THANK YOU
Interested?
DL-PayPal-Cloud-Hiring@ebay.com