Large-Scale Deployments With
Pogo At Yahoo!
Mike Schilli
YAPC::NA 06/14/2012
What’s Pogo?
• Yahoo’s deployment system
– Highly scalable (thousands of targets)
– Highly parallelized rollouts
– In active use for several years
• Open Source version available (pre-alpha)
Pogo in a CI environment
What’s “Deployment”?
• Run one command on many hosts
• Each host “knows” what it’s supposed to look
like via a central database
• One command brings host into “desired” state
• Fetches package updates from edge-server
repos
Run one command on many hosts
• Act on all hosts immediately
Run one command on many hosts
• Act on all hosts immediately
Example Server Farm
Pogo Features
•
•
•
•
•
Sequences
Parallelizes with constraints
Halts if errors violate constraints
Runs health checks (pre/post hooks)
Command agnostic (rpm, apt-get, custom,
etc.)
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Pogo Configuration
Run a Pogo job (tags)
Pogo UI – Single Job View
Pogo UI Overview
Pogo UI – Host View
Pogo UI – Start it Up
$ perl -Ilib bin/pogo-api
pogo-api-28> Listening on 0.0.0.0:7657
$ perl -Ilib bin/pogo-ui -v
pogo-ui-22> Listening on port 5000
Run a Pogo job (targets)
Pogo Configuration
State of Pogo
• Pre-alpha Open Source version on Github
• OSS Pogo will be completed in the coming
months
Pogo on Github
• Developed in the open:
– https://github.com/ytoolshed/pogo
• Internally used with plugins to adapt to
specific data sources
CI Tests with travis-ci.org
The Gory Details
Pogo – User View
Pogo Workflow
Authentication/Authorization
• Web server authentication for submitting jobs
• Target authentication via passwords or
priv/pub keys
• Credentials encrypted with worker pubkey
• Credentials never stored persistently (or
ZooKeeper)
Pogo “Hooks”
• Scripts that run before or after Pogo
commands
• Used to prepare host, perform check health,
or other functions
• Installed locally on target hosts
• rc.d style invocation: run in alphanumeric
order
Pogo “Hooks” (cont.)
• Two types:
– pre hook: run before the specified command
– post hook: run after the specified command
• If any hook script fails:
– No further scripts or commands run
– Host is marked as failed
Pre-hook Examples
• Take a host out of rotation
• Drain database connections
• Check that host is ready to be updated
Post-hook Examples
• Check whether local webserver is serving
expected content
• Ping required services
• Put host back into rotation
Host Failures and Concurrency
• If a host fails:
– Command returns non-zero
– Pre- or post-hook returns non-zero
• The failed host counts against the acceptable
number/percent of unavailable hosts
• Example:
– Four hosts in an app can run at a time
– One host in the app fails
– Only three hosts will be acted on at a time going
forward
Implementation
• AnyEvent Framework
• Asynchronous, event-based
• Added Object::Event for event pub/sub
HTTP Client in AnyEvent
HTTP Client in AnyEvent
HTTP Client in AnyEvent
AnyEvent and Object::Event
AnyEvent and Object::Event
AnyEvent and Object::Event
Queue Processor with AnyEvent and
Object::Event
AnyEvent
bin/pogo-dispatcher
Testing AnyEvent Components
All Pogo Components
All-In-One Pogo: pogo-one
All-In-One Pogo: pogo-one
All-In-One Pogo: pogo-one
Questions?
Thanks!
• Pogo on Github:
– http://github.com/ytoolshed/pogo
Lessons learned along the Way
Set up Unit Tests with Travis-ci.org
Runs immediately after Github
checkins
CI Tests with travis-ci.org
Configure CI with travis-ci.org
Select Github project
Authorize travis-ci.org
Travis-ci.org API Calls
Filling in Passwords
Filling in Passwords
•
•
•
•
Test logging into machines without sshkeys
Convenient for testing
Answer to sudo prompts on targets
Answer to prompts for encrypted packages
Filling in Passwords
• Good password prompt:
mschilli@localhost's password:
• Not password prompt!
… user ''@'localhost' (using password: NO)
PasswordMonkey on CPAN
Plack
Plack
Plack/PSGI
• Plack used for server-independent web
application
– API
– Dispatcher Control Port
Alternatives
•
•
•
•
Capistrano
Rundeck
Aegir
Drush
Pogo UI – Start it Up
$ perl -Ilib bin/pogo-api
pogo-api-28> Listening on 0.0.0.0:7657
$ perl -Ilib bin/pogo-ui -v
pogo-ui-22> Listening on port 5000
Pogo UI
• All Cient Side-Javascript
• Server just redirects to index.html
– Except assets/* js/*
Thanks!
• Pogo on Github:
– http://github.com/ytoolshed/pogo
• PasswordMonkey CPAN Module:
– http://search.cpan.org/dist/PasswordMonkey/
• Travis CI Service:
– http://travis-ci.org