Deploying Lots of Telco
Clouds
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
Introduction
-
Introduction to AT&T
Introduction to the Presenters
Outline:
1.
2.
3.
4.
From loading dock to cloud
How to hatch lots of clouds
How to manage lots of clouds
Lessons learned and Q&A
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
Telco clouds and our challenges
●
What are the Telco Cloud Challenges
Lots of locations
Complex Networks
Service Provider Uptime Requirements
NFV Requirements
Site configurations and the hardware in them can fluctuate
Many enterprise services and global network controllers need to be plugged in and deployed
Build many sites in a compressed time frame
●
Introducing AT&T’s AIC OpenStack Cloud
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
Going From Data Center Loading Dock to Working Cloud
DESIGN
MANAGE
DEPLOYMENT
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
DESIGN
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
Automating Design
What were we trying to fix?
Excel Spreadsheets
Data Entry Mistakes
Inventory updates via e-mail
A long and manual process
No single view
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
AIC Formation
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
AIC Formation - Zone info tab
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
AIC Formation - Rack Elevation Tab
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
AIC Formation - Cable Map
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
AIC Formation - Zone Summary
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
DEPLOYMENT
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
Bootstrap
AIC Formation contains all of the information that we need for site configuration,
like:
●
●
●
Physical device information
Network configuration
Site config version
We feed this information to a simple bootstrap image based on debian minimal
installer, from there the installer sets up a single foundation node with simple
foundational components.
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
Undercloud
Simple undercloud with 4-5 foundational components (containers + KVM):
●
●
●
●
●
PXE Dust
MaaS
Fuel
Jump/Access host
vCenter (in certain sites)
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
Bare-metal conundrum
In our undercloud, we have three different
pieces of software that want to provision baremetal:
●
●
●
OpenStack Fuel
Metal-as-a-Service
vCenter/Auto-deploy
Rather than move the bare-metal provisioning
functionality to one service, we chose to
enable all three.
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
PXE Dust
A very simple solution was to
implement a sort of PXE
router. We did this by:
●
●
Writing a simple python
REST API that allows for
rules and boot targets to
be set
Another API call applies
rules and computes a
boot target
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
OpenStack Fuel
As mentioned in previous slides, OpenStack Fuel was primarily used for installing
OpenStack and non-OpenStack components.
However, there were several scenarios that Fuel was not built for:
●
●
●
Datacenter racks in L3 islands
Micro-roles (AIC has almost 3x the roles of stock Fuel)
Many configuration options are immutable
AT&T added several enhancements to deal with these problems some of which
are found in Mitaka, more that will be seen in Newton and beyond.
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
OpsSimple
●
●
●
●
●
●
●
●
Ansible based
Central config repo
Calls Fuel APIs
Calls MaaS and Libvirt APIs
Executes some commands on nodes for pre and post deployment tasks
Hooks into OSS and BSS systems
All site config and actions are checked into/out of a central git repo
Logical configs (VMs, pseudo net interfaces, etc) sent back to Formation.
Challenge: How to do this with Heat? (Without a full blown OpenStack undercloud) Murano maybe?
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
MANAGE
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
Independent Sites but Centralized Management
AT&T’s massive scale demanded independent zones for performance, resiliency,
and security.
This presented challenges:
-
User Experience Consistency (what are my flavors)
Account Creation (who goes where and what quotas)
Upgrades (orchestrating control plane upgrades)
Tenant Resource Discovery (what can I access)
Operational Challenge
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
Resource Creation Gateway
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
Discovery API Services
Tenant
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
Lessons Learned - Future plans
-
Work upstream when developing customizations to Fuel, MaaS, etc
You can’t invest too much in CI/CD
-
A jenkins/gerrit with deep testing is one of the foundations of our success
-
Overlapping functionalities are okay, but be prescriptive/restrictive with your
teams or they will use things in ways you did not intend.
-
Always more opportunities for automation - going to do it twice?
Automate it!
-
-
This includes monitoring, config management, incident management, network setup, data center physical
deployment (robots), image building, platform definition, etc.
Think machine learning, deep insights, reporting
Let the people do the thinking and the automation do the repetitive work
Infrastructure elements are NOT ISLANDS! We
continue to transform our silo’d teams into crossskilled, interactive and collaborative teams.
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
Thank You!
http://about.att.com/innovationblog/att_cloud_journey
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.
QUESTIONS?
Alan Meadows (am240k@att.com)
Lee Rivere (lr548m@att.com)
Mike Wilson (mwilson@mirantis.com)
© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated
companies. All other marks are the property of their respective owners.