Securing OpenStack Chris C. Kemp About Chris C. Kemp • OpenStack Co-Founder • Former CTO for IT, NASA • Founder and CEO of Nebula, Inc. Chris C. Kemp / Twitter @Kemp Open source software for building private and public clouds. OpenStack is a wonderful choice for the security-minded enterprise’s private cloud *if* best practices are followed during all stages of implementation and operation. I’m not *only* making outrageous claims, I’m going to make some points to back this up with the rest of my talk. OpenStack is a *true* cloud platform 1. 2. 3. 4. 5. On-demand … through self-service interfaces Elastic … dynamically scale up and down Shared … pooled resources Metered by use … at high level of granularity Accessible … broadly over the network OpenStack Details • Multi-tenant, massively scalable, open source cloud operating system. • Supports various hypervisors, including: Xen/XenServer , KVM, Hyper-V, VMWare/ESX, Linux Containers (LXC), QEMU, UML • Flexible network and storage options • Apache 2.0 open source license Why Build Private Cloud? 1. Maturity. Ability to overcome barriers to entry related to culture, process, technology, experience, and tools. 2. Performance. Need to deploy an application near the data and services that are already deployed on premises - lower latency and increased bandwidth. 3. Security. Must keep data inside Company’s security perimeter, where we trust security team, tools, and processes. 4. Cost. TCO much higher for predictable IaaS workloads. 5. Architectural Constraints. Application is not architected to run well in public cloud, or has unique technical requirements. By the Numbers • Community includes 2300 people from 153 Companies • Over 100 active committers with 250K lines of code Who is Involved? And many more… OpenStack Conceptual Architecture So, now we know a little something about OpenStack.. …and we’re forming some initial opinions…. CIO “Sounds exciting!” CSO “Sounds target-rich..” Security and the OpenStack project OpenStack Security Community Highlights OpenStack project groups – – – – Vulnerability response Formalizing security Audit projects Multiple security-centric blueprints – ongoing code improvements Commercial efforts – Professional penetration Testing / API fuzzing – Sponsored bugfests with growing participation – Active and ongoing source code review process Bringing defense in depth to OpenStack clouds Assuming you’ve laid the foundation…. Before we begin we’ll need the governance, guidance, and groundwork that will define the requirements.. • Compliance and Audit • ERM • Legal and Electronic Discovery • Information Lifecycle • Corporate policy • Portability • Interoperability • Architecture • Operations “touch points” ….let’s take a closer look at the technologies in play. OpenStack Conceptual Architecture OpenStack Logical Architecture OpenStack - Under the hood… • Mostly implemented in Python • REST and WSGI communication between services • Multiple application choices to implement backend • databases • queue • networking Everything needs to be hardened and continuously monitored. …Luckily, we have a few best practices for doing this stuff with open source software. OpenStack compute service (nova) • Equivalent to Amazon EC2 • Runs virtual machines on hypervisor of your choice • Includes support for block volumes external to hypervisor • The architecture of nova allows for massive parallel scaling, but to get there requires some complexity. Underlying technologies • nova-compute – speaks to libvirt, XenAPI, etc. • nova DB – SQLalchemy & a SQL DB • queue – Python Kombu+AMQPlib – rabbitmq Underlying technologies • Nova-network – Provides connectivity • Nova-volume – provides volume API – Backended with iscsi, sheepdog, ceph, etc • nova-scheduler – Determines available resources – Assigns workloads OpenStack Compute Security Considerations • Secure your hypervisor – This is a topic for another talk… but certainly not trivial. • Choose a database – Consider high availability – Enhanced security configuration • Message Queue – Harden the queue software’s configuration – Monitor and correlate queue messages • Choose filesystem(s) – Enable filesystem’s security features – Deploy hardened daemons – Monitor activity • Monitor API access OpenStack object store (swift) • Similar to Amazon S3 • Configurable number of duplicate object replicas • Supports geo-replication of objects • Internally: – memcache provides caching for scale and speed – SQLite – rsync – python greenlets/eventlet OpenStack Object Store Security considerations • Properly secure underlying technologies – memcached, rsync • Implement and test RBAC • Restrict admin read access to objects – Least privilege, is admin read access required? • Integrated information lifecycle – Automate / integrate IL processes when possible • Monitor & correlate API access – Record all access to the object store OpenStack Dashboard (Horizon) • Standard webapp stuff • django-based • Uses keystone for authN/Z OpenStack Dashboard Security considerations • Use enterprise authentication behind keystone • Standard webapp hardening process • Protect credentials • Monitor access and correlate OpenStack image service (glance) Provides a repository for VM images and snapshots • SQL for metadata • Supports for multiple backend filesystems • Ceph • S3 / Swift • Local FS OpenStack Image Service Security Considerations • Choose distributed filesystem(s) – Enable filesystem’s security features and configure hardened endpoints – Monitor activity • Choose a database – Consider high availability – Enhanced security configuration • Audit – Automate audit of images for OS controls • Patch management – Automate patching and configuration updates to OS images OpenStack Identity Service (keystone) • authN and authZ provider for OpenStack • Rewrite introduced a new architecture – Straightforward integration with commercial / external auth products and solutions OpenStack Identity Service Security considerations • Use backend enterprise authentication provider – – OpenStack is not an identity project – Keystone’s backend API provides easy integration for authN, and acceptable authZ • Monitor API access – Attempts – Failures • Logging - Monitor and correlate – Monitor identities across OpenStack – Debug loglevel is informative but sensitive Other parts of the OpenStack ecosystem • OpenStack incubated projects – Two exciting networking projects • Quantum • Mélange • Other interesting OpenStack projects – Database-as-a-Service – Dashboard enhancements and plugins – Hybrid cloud functionality (cloudgateway, etc) …Zooming back out Enough of the trenches. This is a keynote, after all. OpenStack seems to be made up of defensible technologies – – – – – – – – – Lots of readable python Databases: sqlite, mysql, postgres Message queue: rabbitMQ Distributed Filesystems: gluster, ceph Hypervisors: Xen, KVM, ESXi, Hyper-V* memcached django authN / authZ API interface Linux security features OpenStack Logical Architecture …But It’s the responsibility of the implementer to turn the “security switches” to “on.” So - OpenStack isn’t a production-ready cloud? • Most technical security controls required for compliance are NOT built in to OpenStack. • • That shouldn’t dissuade you. • The building blocks are all in place. Monitor Harden Integrate Hardening OpenStack system environments • Restrict network and data access to least privilege • Enable security features of underlying software • Configure security features of the underlying OS • Harden the hypervisor • Use PKI for SSL • Implement database security Integration - benefits to even the playing field Some integration points: • • • • • • SIM/ SIEM IR automation / Live Forensics CMDB / Service desk Asset mgmt / Patch mgmt Auditing process automation IPAM Integrating the underlying cloud framework into these elements yields huge benefits Monitoring – Benefits in the open cloud OpenStack is powerful foundation to build advanced security controls Building complex solutions becomes relatively simple • • • • SIEM sees significant benefits Automated Incident Response Cloud-wide flow monitoring Security appliances: IPS-aaS, FW-aaS, … Defense in depth of workloads in cloud • An integrated defense in depth strategy can benefit from open source software and from private cloud • OpenStack is a great example Back out to “The big picture” - CSA Security Guidance - Critical Areas of Focus in Cloud Mapping the Cloud Model to the Security Control & Compliance Model Looking ahead - 2012 and OpenStack security The coming year looks to be very exciting for the OpenStack project, and specifically for OpenStack security. OpenStack-based products could offer powerful security options. A few ideas we’re kicking around at : • Interesting security-as-a-service potential – Quantum provides some of the missing building blocks needed for metered and scalable security controls on demand in OpenStack • IR process integration offers excellent coverage – Potential for huge efficiency improvements in remediation of incidents and live response activities • SIM / SIEM benefits – Coverage over large infrastructure increases value of SIM integration – The visibility and control that IaaS offers eases SIM complexity Conclusion • OpenStack is a flexible foundation – It’s a viable option, but not necessarily right out of the box – It’s not right for every workload or enterprise – Its open-ness is a big plus for security • Still some significant unanswered security questions – Expect to see commercial OpenStack-based products bridging this gap • Exciting new developments improving the security of OpenStack are happening every day Thanks for listening! Chris C. Kemp Twitter: @Kemp