Vac, Vac-in-a-Box, and
Vcycle
Andrew McNab
University of Manchester
LHCb
Overview
●
Vac and Vcycle status
●
Most news is about Vac
●
●
Introduction to Vac-in-a-Box
●
Containers and CernVM
●
Plans
Observations about OpenStack
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
2
Vac and Vcycle
●
Two GridPP systems aimed at running VMs
●
Vac - autonomous hypervisors
●
●
Each factory machine creates VMs in response to
observed demand for each type of VM
Vcycle - uses OpenStack etc
●
●
●
Factories created via Cloud API in response to observed
demand for each type of VM
OCCI (EGI) and Azure (MS) plugins from CERN
Now also working with commercial OpenStack provider
(DataCentred at MediaCity)
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
3
Vac, Vcycle, HTCondor Vacuum deployment
Vac
Vcycle
ATLAS
CMS
LHCb
GridPP DIRAC
Manchester
✔
✔
✔
✔
Oxford
✔
✔
✔
✔
Lancaster
✔
✔
✔
UCL
✔
✔
CERN (LHCb)
✔
CERN (Dev)
✔
✔
✔
✔
Imperial
✔
✔
✔
✔
CC-IN2P3
✔
DataCentred
✔
✔
✔
✔
HTCondor Vacuum at RAL
✔
✔
4
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
Vac installation and support
●
Detailed Admin Guide in distribution and on website
●
●
There is a Support Unit for Vac/Vcycle in GGUS
●
●
And a service type for Vac in GOCDB
The Vac distribution includes a Puppet module
●
●
Plus mailing list and other website contents
So it can be managed like a conventional service or piece of
grid middleware
Aim is to make the Vac daemon easy for system admins to manage
●
●
Factory machines are autonomous so less worrying about small
failures of services that have big consequences
man pages, Vac rereads config files each ~minute, config
merger of /etc/vac.conf and /etc/vac.d/*.conf
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
5
Vac site
V VVVV
HEP-specific software
is hidden inside the VMs
apart from Vac.
Vac factory
V VVVV
VM
VM
VM
VM
V VVVV
VM
V VVVV
vacd daemon
V VVVV
V VVVV
puppet client
PXE BIOS
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
6
Vac site
V VVVV
HEP-specific software
is hidden inside the VMs
apart from Vac.
But still need to run
the usual set of
Linux site services.
DNS service
DHCP service
Vac factory
V VVVV
VM
VM
VM
VM
V VVVV
VM
V VVVV
vacd daemon
Puppet server
V VVVV
V VVVV
TFTP service
puppet client
PXE BIOS
Squid server
Kickstart /
HTTP server
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
7
Even easier: Vac-in-a-Box
●
●
●
Vac-in-a-Box (ViaB) aims to make it ridiculously easy to
install Vac-based sites
Extend the autonomous factory machine model to managing
the operating system too
Remove all reliance on central service machines at the site
●
●
●
But be flexible to accommodate site policy about DHCP etc
Each factory not only runs Vac to create machines, but also
●
A Squid cache used by CernVM-FS in the VMs
●
All the services for any other machine to install itself
Configuration is managed through viab.gridpp.ac.uk website
●
Files fetched from website are static, so easy to scale up
multiple instances of the web server
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
8
Vac-in-a-Box website
●
Configuration is managed by site and/or regional admins
●
●
Only required when changes are needed
Right to modify settings based on per-site lists of DNs
●
Intend to pull these from GOCDB lists of site ops
●
Sites use GOCDB-style names like UKI-NORTHGRID-MAN-HEP
●
Sites consist of one or more “spaces”, equivalent to CEs
●
In Vac:
●
Spaces consist of factory machines, with IP, MAC etc
●
Spaces have one or more vmtypes
●
details of how to contextualise that type of VM
●
and its target share of the space
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
9
Site configuration
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
10
Space configuration
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
11
Host (factory) configuration
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
12
Vmtype configuration
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
13
Vac-in-a-Box docs
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
14
DNS and DHCP in ViaB
●
Since we have a list of factories including IP/DNS, can provide
DNS within ViaB
●
●
●
Can use site DNS servers for general DNS queries or generic
DNS like 8.8.8.8/8.8.4.4
Each factory runs a DHCP server
●
Again based on the configured list of IP/DNS/MAC
●
Set non-authoritative, so can co-exist with site DHCP
●
●
Instead of running private DNS services, ViaB just lists each
factory in /etc/hosts on each factory
If necessary, site DHCP can be configured to respond for
factories instead
All this means you can run a ViaB site without your own DNS/
DHCP servers, or getting changes to them if someone else does
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
15
Squid caches
●
Need access to Squid HTTP caches for CernVM-FS
●
●
VMs use CernVM, which gets OS via CernVM-FS too
Site Squids can be single points of failure, performance
bottlenecks etc
●
ViaB puts a Squid cache on each factory machine
●
Used for VM CernVM-FS and RPM installs on that machine
●
Squids use ICP UDP protocol to check if any other Squid in
the space has the requested file in their cache
●
●
Only then is an upstream or direct request made
Avoids need for site Squid(s). Very robust.
●
Least loaded Squid will typically reply first via ICP.
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
16
PXE installation in ViaB
●
●
●
●
A new factory boots with its PXE BIOS and uses a DHCP reply from one
other ViaB machine (the “responder”)
The responder DHCP tells PXE BIOS on installing machine to contact
the responder’s TFTP too
TFTP on each ViaB factory has the usual install files, plus a pointer to
use the ViaB Kickstart file from viab.gridpp.ac.uk
Kickstart file tells Anaconda installer to use the responder’s Squid
cache for fetching the SL6, CA, APEL etc RPMs
●
The site’s viab-conf RPM is fetched from viab.gridpp.ac.uk
●
Installation proceeds as usual, taking ~5 minutes
●
The machine boots straight into SL6 using kexec, so no PXE boot
looping
●
●
A reboot forces a reinstall, unless you manually skip PXE
To get the first factory up, a USB boot image is provided
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
17
Vac-in-a-Box configuration management
●
●
●
Configuration of the site is managed on viab.gridpp.ac.uk
Each time the “Publish” button is used, a new viab-conf RPM is generated
and put in the site-specific YUM repo
●
Includes a list of factory IP/DNS/MAC
●
vmtype configurations for Vac
●
X.509 certificates and encrypted keys in .p12
●
SSH authorized_keys list
●
Idempotent viab-conf-postinstall to apply any changes
●
RPM dependencies to install more RPMs via YUM
A persistent partition at /etc/viabkeys has the .p12 password
●
●
Has to be set up once per factory, but survives reinstalls
YUM auto updates are enabled to ensure security updates are applied
●
Less risk of breaking everything since no middleware other than APEL
●
All YUM updates other than viab-conf are done via factory Squid caches
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
18
Vac+Puppet site
HEP-specific software
is hidden inside the VMs
apart from Vac.
But still need to run
the usual set of
Linux site services.
V VVVV
DNS service
DHCP service
Vac factory
V VVVV
VM
VM
VM
VM
V VVVV
VM
V VVVV
vacd daemon
Puppet server
V VVVV
V VVVV
TFTP service
puppet client
PXE BIOS
Squid server
Kickstart /
HTTP server
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
19
Vac-in-a-Box site
HEP-specific software
is hidden inside the VMs
apart from Vac.
V VVVV
Vac factory
V VVVV
Site services are
hidden inside the Vac
factory machines.
VM
VM
VM
VM
V VVVV
VM
V VVVV
vacd daemon
V VVVV
V VVVV
YUM+viab-conf
DHCP
Squid
hosts
TFTP
viab.gridpp.ac.uk
PXE BIOS
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
20
Back to generic Vac …
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
21
Containers
●
●
CernVM group now offering a technology preview of
container-based machines using Docker
Plan is to use this to offer containers managed by Vac
●
●
●
Will run CernVM-FS on the factory
So a trade-off in managing that vs advantages of
containers
So where I say “virtual machine” in these slides, I could
say “logical machine” to be general
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
22
Vac next steps
●
New VacQuery UDP protocol to scale up beyond ~100
factories in the future
●
Container support
●
Fetch ops DNs for viab.gridpp.ac.uk from GOCDB
●
Provide an (even more) OpenStack-like environment for the
VMs from Vac
●
Integrate Squid-on-factory mode in Vac Puppet module
●
… but stay near current codebase of ~3300 lines of Python
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
23
OpenStack codebase commits
“CY15-Q1 Community Analysis — OpenStack vs OpenNebula vs Eucalyptus vs CloudStack”
http://www.qyjohn.net/?p=3801
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
24
Summary
●
Vac and Vcycle continue to be in use by a matrix of
sites and experiments
●
Vac-in-a-Box now further simplifies creating a Vac site
●
More Vac and Vcycle development in the pipeline
●
OpenStack appears to be the clear winner among
Open Source cloud implementations
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
25
Target shares: ATLAS vs LHCb
Each autonomous Vac
machine uses VacQuery
UDP protocol to discover
what else is happening at
the site.
Compares this against
target share for each type
of VM (~1 per
experiment).
Creates new VMs for
experiments currently
under their share.
But backs-off creating
types of VM which are
failing to find any work to
do.
26
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
Example of Vac configuration
●
Section of vac.conf used to enable LHCb VMs at Manchester
●
They just need this and to create a hostcert/key.pem
●
Compare what YAIM has to do to add a VO to a CE/Batch site
[vmtype lhcbprod]
vm_model = cernvm3
root_image = https://lhcbproject.web.cern.ch/lhcbproject/Operations/VM/cernvm3.iso
rootpublickey = /root/.ssh/id_rsa.pub
backoff_seconds = 600
fizzle_seconds = 600
max_wallclock_seconds = 172800
log_machineoutputs = True
accounting_fqan=/lhcb/Role=NULL/Capability=NULL
heartbeat_file = vm-heartbeat
heartbeat_seconds = 600
user_data = https://lhcbproject.web.cern.ch/lhcbproject/Operations/VM/user_data
user_data_option_dirac_site = VAC.Manchester.uk
user_data_option_cvmfs_proxy = http://squid-cache.tier2.hep.manchester.ac.uk:3128
user_data_file_hostcert = hostcert.pem
user_data_file_hostkey = hostkey.pem
27
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
Vcycle
Apply Vac ideas to
OpenStack etc
Cloud Site
Site
Vcycle
VM factory
s
uest
Req al jobs
e
for r
Third party
Vcycle
VM factory
Central
agents &
services
Vcycle implements an
external VM factory
that manages VMs.
Pilot VM. Runs
Job Agent to
fetch from TQ
Vcycle
VM factory
No direct
communication
between Vcycle
and task queue
Matcher &
Task Queue
Can be run centrally
by experiment
or by site itself
or by a third party
VMs started and
monitored by Vcycle, but
not managed in detail
(“black boxes”)
User and
production
jobs
Vac, Vac-in-a-Box, Vcycle - Andrew.McNab@cern.ch - GridPP35, Sep 2015, Liverpool
28