Benoît Créau
Aix System Engineer
@chmod666
chmod666org
http://chmod666.org
benoit.creau@chmod666.org
Using Chef Automation on AIX
© Copyright IBM Corporation 2015
Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.
9.0
whoami
• AIX/PowerVM System Engineer working in Large french banks
• Passionate about AIX and Power Systems
• Blogger since Feb 2010
– http://chmod666.org
• Creating tools for AIX an Power
– Github : chmod666org
• Sharing my daily life as a sysadmin on Twitter (tips and tricks,
blog post)
– @chmod666
• Recently Redbook author: PowerVC 1.2.3 Introduction and
configuration
(www.redbooks.ibm.com/redpieces/abstracts/sg248199.html?
Open)
© Copyright IBM Corporation 2015
2
What is Chef ?
• Automation platform (like Ansible, Puppet, Saltstack)
• Automate how systems/applications (we’ll talk here about
systems only) are:
– Deployed (ie. build time)
– Configured (ie. post-install time)
– Managed (the life of the system after its installation)
• Client/Server architecture:
– Chef Server : available only on x86 (https://downloads.chef.io/chef-server/)
– Chef Client : AIX Client available (https://downloads.chef.io/chef-client/aix/)
• Based on Ruby (You’ll have to learn Ruby if you want to
develop your own providers) (don’t worry ruby=super simple)
• Available for free (without premium features)
• Hosted server and support available provided by
Chef/Opscode (https://www.chef.io/chef/)
© Copyright IBM Corporation 2015
Why using Chef ?
• It’s the only automation tool with a supported client available
on AIX (https://www.chef.io/blog/2014/12/08/announcing-chefclient-for-ibm-aix/)
– Available since December 8, 2014
• IBM and Chef work together ! (ie. have you already tried to
install IBM Cloud Manager with Openstack, it’s Chef based !)
• As we need to create provider,modules (or whatever you call
them), I think in my humble opinion that there is no place for
two automation tools (ie. there is a war between Puppet, and
Chef)
• We are already working on a complete AIX cookbook ! AND
WE ARE MOVING FAST !
© Copyright IBM Corporation 2015
Advantages ?
• Automation (build, deploy,
manage)
• Your infrastructure becomes:
– Versionable (chef-repo build over a
git repo)
– Testable
– Repeatable
• Chef turns infrastructure into
code
• Use it for:
– Patch Management
– Post-installation
– Configuration management
© Copyright IBM Corporation 2015
Terms 1/2
• A few things you need to know before beginning:
– Cookbook: A bunch of recipes, provider and templates
– Recipe: Describe what and how to do things (install a fileset, mount a
filesystem)
– Provider/resource: The piece of code that configure an item (create a
directory, changing a device attribute …)
– Node: where the chef-client is run
– Server: the chef server holds all nodes configuration data
(cookbooks,recipes,templates)
– Environment: production, test, dev, site, ….
– Templates: Used to generate static texts files (contains statements
and expressions)
– Roles: database, application, webserver
© Copyright IBM Corporation 2015
Terms 2/2
• What’s the idempotence ?
– Running the tool over and over doesn't change the system after the
first time.
– For instance if you create a directory in a recipe if the directory
already exists chef will not try to recreate it.
• What’s the convergence ?
– A recipe converges when you do an action (creating the directory: it
converge, not creating it: it does not converge)
© Copyright IBM Corporation 2015
chef-client installation 1/3
• Download here: http://tinyurl.com/ChefAIX12-5-1-1
• Check for new versions. There are new release very often!
• 87.7MB (it’s huge)
• Use 6.1 version for both AIX 6.1 and AIX 7.1
• Don’t follow Chef website instructions
# installp -ld chef-12.5.1-1.powerpc.bff
Fileset Name
Level
I/U Q Content
====================================================================
chef
12.5.1.1
I N usr,root
© Copyright IBM Corporation 2015
chef-client installation 2/3
• The name of the package is « chef »
# installp -aXYgd . chef
+-----------------------------------------------------------------------------+
Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...
SUCCESSES
--------Filesets listed in this section passed pre-installation verification
and will be installed.
Selected Filesets
----------------chef 12.5.1.1
# The full stack of chef
<< End of Success Section >>
+-----------------------------------------------------------------------------+
Installing Software...
+-----------------------------------------------------------------------------+
installp: APPLYING software for:
chef 12.5.1.1
Restoring files, please wait.
13149 files restored.
Thank you for installing Chef!
Finished processing all filesets.
13149 files ! Whooo !
(Total time:
1 mins 38 secs)
© Copyright IBM Corporation 2015
Chef-client installation 3/3
• Install any utf8 english language environment (not mandatory
but recommended to avoid any warning messages)
# smitty lang
© Copyright IBM Corporation 2015
What’s in the fileset ?
• chef-client
• chef-solo
• Ohai
• Ruby (chef is based on Ruby)
• Some « default » providers:
–
–
–
–
cron
group
mount
services
– inittab
# find /opt -type f | grep provider | grep aix
/opt/chef/embedded/apps/chef/lib/chef/provider/cron/aix.rb
/opt/chef/embedded/apps/chef/lib/chef/provider/group/aix.rb
/opt/chef/embedded/apps/chef/lib/chef/provider/ifconfig/aix.rb
/opt/chef/embedded/apps/chef/lib/chef/provider/mount/aix.rb
/opt/chef/embedded/apps/chef/lib/chef/provider/package/aix.rb
/opt/chef/embedded/apps/chef/lib/chef/provider/service/aix.rb
/opt/chef/embedded/apps/chef/lib/chef/provider/service/aixinit.rb
/opt/chef/embedded/apps/chef/lib/chef/provider/user/aix.rb
© Copyright IBM Corporation 2015
Begining on Chef chef-solo 1/2
• Before trying to work with chef-server you can start to learn
Chef with chef-solo, it’s the first thing to do!
• You don’t need a chef-server to do that, just install the client on
an AIX box.
• Configuration:
– Create directories:
# mkdir –p ~/chef/cookbooks/aix/providers
# mkdir –p ~/chef/cookbooks/aix/recipes
# mkdir –p ~/chef/cookbooks/aix/resources
– Create solo.rb:
# cat solo.rb
file_cache_path "/root/chef"
cookbook_path "/root/chef/cookbooks"
json_attribs "/root/chef/node.json"
– Create node.json:
# cat node.json
{
"run_list": [ "recipe[aix]"]
}
Where are my
cookbooks
Run all the recipes in
the cookbook called
« aix »
© Copyright IBM Corporation 2015
Describes the
« run_list »
Begining on Chef chef-solo 2/2
• Running chef-solo:
# chef-solo -c ~/chef/solo.rb
• You will need to debug you recipes and provider:
– Run chef-solo in info mode (will print all Chef::Log.info)
# chef-solo -c ~/chef/solo.rb –l info
– Run chef-solo in debug mode (will print all Chef::Log.debug)
# chef-solo -c ~/chef/solo.rb –l debug
• Running particular recipes:
# chef-solo -c ~/chef/solo.rb –o
“recipe[aix::install_filesets],recipe[aix::install_ssh]”
Will only run recipes called « install_filesets » and « install_ssh »
• Running particular environment:
# chef-solo -c ~/chef/solo.rb –E production OR # chef-solo -c
~/chef/solo.rb –E oracle
© Copyright IBM Corporation 2015
Writing your first recipe ! Hello world ! 1/4
• Recipes are run from the top down:
# cat ~/chef/cookbook/aix/recipes/hello_world.rb
directory “/tmp/hello_ibm_techu” do
owner ‘root’
group ‘system’
mode ‘0755’
action :create
end
file “/tmp/hello_ibm_techu/hello” do
content “Hello IBM Technical university ! Chef
is AWESOME!”
end
• 1 – We are creating a directory /tmp/hello_ibm_techu
• 2 – We are writing a text in a file called hello in this directory
© Copyright IBM Corporation 2015
Writing your first recipe ! Hello world ! 2/4
• Running this recipe:
Creating the directory
Writing the file
2 resources ok !
© Copyright IBM Corporation 2015
Writing your first recipe ! Hello world ! 3/4
• Does it work ?
• Rerun it (idempotence !!!!!)
Already ok ! Do nothing !
0 resources !
© Copyright IBM Corporation 2015
Writing your first recipe ! Hello world ! 4/4
• Let’s change the content of the file
file “/tmp/hello_ibm_techu/hello” do
content “Tyrell : Is this to be an empathy test? Capillary
dilation of the so-called blush response? Fluctuation of the
pupil. Involuntary dilation of the iris...
Deckard: We call it Voight-Kampff for short.”
end
Removed !
1 resource !
© Copyright IBM Corporation 2015
Added!
Ohai 1/2
• What’s ohai ?
– Ohai is a tool that is used to detect attributes on a node, and then
provide these attributes to the chef-client at the start of every chefclient run. Ohai is required by the chef-client and must be present on a
node. (Ohai is installed on a node as part of the chef-client install
process.)
– The types of attributes Ohai collects include (but are not limited to):
• Platform details
• Network usage
• Memory usage
•
•
•
•
CPU data
Kernel data
Host names
Fully qualified domain names
• Other configuration details
• You can create custom plugins
© Copyright IBM Corporation 2015
Ohai 2/2
• Ohai attributes accessible in any recipes, templates (it’s a
tree):
node[‘ipaddress’]
node[‘macaddress’]
node[‘os’]
node[‘os_version’]
node[‘platerform_version’]
node[‘virtualization’][‘lpar_name’]
node[‘virtualization][‘lpar_no’]
node[‘filesystem’][‘/dev/hd4’][‘kb_size’]
node[‘filesystem’][‘/dev/hd4’][‘kb_used]
node[‘filesystem’][‘/dev/hd4’][‘precent_used’]
node[‘filesystem’][‘/dev/hd4’][‘mount’]
node[‘filesystem’][‘/dev/hd4’][‘fs_type’]
© Copyright IBM Corporation 2015
Ohai custom plugins 1/2
• You may want to add your own custom attributes to ohai (in my
case I need the oslevel to do the update of my AIX hosts)
– It’s a piece of ruby code:
– In the example below I’m getting:
• The oslevel
• The nodename
• The partition name
• The memory mode
© Copyright IBM Corporation 2015
Ohai custom plugins 2/2
• Put your plugins in your chef client configuration:
# grep ohai client.rb
Ohai::Config[:plugin_path] << '/etc/chef/ohai_plugins'
• Or run ohai with plugins path:
# ohai -d /etc/chef/ohai_plugins
• Check it works:
• Now access oslevel with: node[‘aixcustom’][‘oslevel’]
© Copyright IBM Corporation 2015
Templates 1/4
• Writing your templates files (ohai attributes)
# cat /root/chef/cookbooks/aix/templates/default/motd.erb
_______
_
_
_
|__
__|
| |
| | | |
| | ___ ___| |__ | | | |
| |/ _ \/ __| '_ \| | | |
| | __/ (__| | | | |__| |
|_|\___|\___|_| |_|\____/
hmc_lpar_name:
<%= node['virtualization']['lpar_name'] %>
hostname:
<%= node['hostname'] %>
hmc_lpar_number: <%= node['virtualization']['lpar_no'] %>
ip_address:
<%= node['ipaddress'] %>
os:
<%= node['os'] %> <%=
node['platform_version'] %>
uptime:
<%= node['uptime'] %>
vendor_id:
<%= node['hostname'] %>
cpu:
<%= node['cpu']['0']['model_name'] %>
fortune:
A new life awaits you in the Off-World colonies!
© Copyright IBM Corporation 2015
Templates 2/4
• The recipe:
template '/etc/motd' do
source 'motd.erb'
end
•
•
•
•
•
•
•
•
lpar_name
hostname
IP address
OS
Uptime
Vendor_id
CPU
….
© Copyright IBM Corporation 2015
Templates 3/4
• Rerun it ? Obviously uptime has changed !!!
Changed !
© Copyright IBM Corporation 2015
Templates 4/4
• Conditions:
<% if node['cpu']['total'] == 1 -%>
This computer has <%= node['cpu']['total'] %> CPU!
<% else -%>
This computer has <%= node['cpu']['total'] %> CPUs!
<% end -%>
• Loops:
<% node['filesystem'].each do |key, value| -%>
fs <%= key %>
<% end %>
© Copyright IBM Corporation 2015
Let’s get serious now !
• You know know the basics ! It’s now time for real life examples
!
© Copyright IBM Corporation 2015
Updating your servers with Chef 1/5
• What do you do when updating ?
–
–
–
–
–
Checking with lppchk there are no broken filesets (lppchk)
Committing all uncomited filesets (installp)
Remove any installed efix (emgr)
Make a backup of your system (alt_disk_copy)
Run the update:
• While doing the alt_disk_copy
– In the case mount a remote directory from your nim server
• Using nimclient
– The nimclient must be configured
• Using multibos
– In this case there is no need to make a backup of your system
– Umount all mounted directory
– Check with lppchk there are no broken filesets
– http://chmod666.org/index.php/updating-tl-and-sp-using-chef/
© Copyright IBM Corporation 2015
Updating you servers with Chef 2/5
• You guys are lucky I have done all the job for you !
– Providers available: multibos, emgr, nimclient, niminit, alt_disk_copy
– The recipes available:
• update_by_multibos
– Do a multibos and update it
• update_by_altdisk
– Do an alt_disk_copy and update it
• update_by_nimclient
– Create an alt_disk_copy (to backup you server)
– Use nimclient to do the update
– The nimclient providers needs a special naming convention of you nim
lpp_source to determine with sp/tl to apply
# nimclient -l -t lpp_source | grep 7100
7100-03-00-0000-lpp_source
resources
7100-03-01-1341-lpp_source
resources
7100-03-02-1412-lpp_source
resources
7100-03-03-1415-lpp_source
resources
7100-03-04-1441-lpp_source
resources
7100-03-05-1524-lpp_source
resources
lpp_source
lpp_source
lpp_source
lpp_source
lpp_source
lpp_source
© Copyright IBM Corporation 2015
Update your servers: update_by_multibos 3/5
Recipe
Execution
© Copyright IBM Corporation 2015
Update your server: update_by_altdisk 4/5
© Copyright IBM Corporation 2015
Update your servers update: by nimclient 5/5
© Copyright IBM Corporation 2015
Batch it ? 1/2
• Push-job ?
• knife-ssh: allows you to run the client directly from ssh (from the knife workstation). Use it to
batch everything you want
– Change motd on all my websphere servers
– Update all my AIX box in 6.1
– …. Anything you can imagine
Node1  Tyrell
Node2  Priss
Node3  Batty
© Copyright IBM Corporation 2015
Batch it 2/3
• Let’s check now with an update ! (update all my servers with the role
« france_testing » to the latest available service pack using nimclient, and make an
alt_disk_copy before !)
• Before update:
• Running update:
© Copyright IBM Corporation 2015
Batch it 3/3
• After update:
• In on single command (on all systems)
– Checking fileset
– Finding disk to make a copy of the system (for backup
purpose)
– Creating nimclient configuration
– Finding the latest service pack
– Updating to this service pack
• Tell me now if any other tools that exists is doing that !
© Copyright IBM Corporation 2015
Execute ! Execute ! Execute ! (lvm case)
• When there is no provider available use the « execute »
resource.
• It’s the case for any lvm action. (We are working on a lvm
provider … help us) (this one is hard)
• ALWAYS USE GUARDS WITH EXECUTE !!!!!
execute “create decakrdvg” do
command “mkvg –y deckardvg –S –s 256 hdisk12”
creates “/dev/deckardvg”
end
execute “create decakrdlv” do
command “mklv -t jfs2 -y deckardlv deckardvg 1G”
creates “/dev/deckardlv”
end
execute “create fs for deckardlv” do
command “crfs –v jfs2 -d deckardlv –m /apps/deckard –A yes”
creates ‘/apps/deckard‘
end
© Copyright IBM Corporation 2015
AIX postinstall with chef
• Super short example:
– Creating root home and .profile.
– Installing ssh and ssdpcm.
– Changing /etc/motd.
Recipe: aix7::ohai_custom
* ohai[reload] action reload
- re-run ohai and merge results into node attributes
* template[/etc/chef/ohai_plugins/aixcustom.rb] action create (up to date)
Recipe: aix7::create_fs_rootvg
* execute[hd3] action runThe filesystem size is already 2097152.
- execute chfs -a size=1024M /tmp
* execute[hd9var] action runThe filesystem size is already 1048576.
- execute chfs -a size=512M /var
* execute[/apps] action run (skipped due to not_if)
* mount[/apps] action mount (up to date)
Recipe: aix7::create_profile_root
* directory[/root] action create (up to date)
* user[root] action create (up to date)
* file[/root/.profile] action create (up to date)
Recipe: aix7::motd
* template[/tmp/motd] action create (up to date)
Recipe: aix7::install_ssh
* bff_package[openssh.license] action install (up to date)
* bff_package[openssh.base] action install
- install version 6.0.0.6103 of package openssh.base
* service[sshd] action start (up to date)
Recipe: aix7::install_sddpcm
* bff_package[devices.fcp.disk.ibm.mpio] action install
- install version 1.0.0.24 of package devices.fcp.disk.ibm.mpio
* bff_package[devices.sddpcm.71.rte] action install (up to date)
© Copyright IBM Corporation 2015
Chef & PowerVC
• Use Chef with PowerVC to
build your new machines!
• Configure cloud-init to setup
the chef-client
• At boot time chef-client will
run and pull all the
recipes/providers needed
• Way better than cloud-init
alone, or custom shell
scripts.
• Look here to start:
http://tinyurl.com/ChefAIXPowerVC-CloudInit/
© Copyright IBM Corporation 2015
Chef and PowerVC: cloud-init
• Put this in /opt/freeware/etc/cloud/cloud.cfg, this will
automatically configure the chef-client and register a new node
on the chef-server
• Then (in the example below) all the recipes from the aix
cookbook will be launched !
chef:
force_install: false
node_name: deckard
server_url: "https://chefserver.lab.chmod666.org/organizations/chmod666"
validation_name: "chmod666-validator"
validation_key: |
-----BEGIN RSA PRIVATE KEY----MIIEpQIBAAKCAQEApj/Qqb+zppWZP+G3e/OA/2FXukNXskV8Z7ygEI9027XC3Jg8
[..]
XCEHzpaBXQbQyLshS4wAIVGxnPtyqXkdDIN5bJwIgLaMTLRSTtjH/WY=
-----END RSA PRIVATE KEY-----
run_list:
- "role[aix]"
runcmd:
- /usr/bin/chef-client
© Copyright IBM Corporation 2015
Chef & PowerVC: Activation input
• Then in the activation input, customize your newly build VM
with different recipes/roles
• A production oracle machine ?
#cloud-config
chef:
run_list:
- “role[oracle]“
environment: “production”
• A developpement websphere machine ?
#cloud-config
chef:
run_list:
- “role[websphere]“
environment: “developpement”
• After writing recipes! It’s that simple !
© Copyright IBM Corporation 2015
What is missing ?
• Some providers (check next slides)
• Push-jobs client for AIX (instead of using knife-ssh, another
way is to use push-jobs to run chef-client from a single point of
management)
• Chef-server:
– Unfortunately not running on AIX.
– Not running on Linux on P (really really wants to see chef server
running on Linux on P)
• Recipes for application and middleware. It think it will be worth
it to write recipes for orcale, websphere, tsm and all major third
parties tools running on AIX
© Copyright IBM Corporation 2015
The Chef AIX Cookbook: current status
• We are moving fast there are more and more providers
available in the current AIX cookbook
• I’m one of the official maintener/contributor on this cookbook. (I
have the authority to merge your pull request)
• https://github.com/chef-cookbooks/aix
• Current available providers:
– inittab
– no, chdev, chsec, tunables
– services
– fixes (emgr)
– nimclient/niminit
– alternate disks (alt_disk_copy, alt_rootvg_op)
– /etc/hosts (hostent)
– bootlist
– subserver
– Tcpservices
– Multibos
– For other this use execute or wait (or code ) for the provider !
© Copyright IBM Corporation 2015
We need your help !
• We need your help, even testing recipes, providers is useful:
–
–
–
–
–
–
Write providers
Write recipes
Test our providers
Test our recipes
Give us ideas
We have a list of not finished providers:
• Dump devices
• Wpars
• Nim (server side)
• Virtual I/O Servers
• LVM (we are on it so don’t try to code this one, especially if you are a beginner)
• ODM
• System resources controller (mkssys, rmssys, …)
• Swap
• Rbac, ldap
• System backups (mksysb)
© Copyright IBM Corporation 2015
Chef : something bigger?
• Chef is part of something bigger.
• Don’t start this alone, talk to your colleagues and decide
together what’s the best way to begin !
• Talk to other teams, you have to adopt a common strategy
– DBA: write recipes and provider to create and manage new databases
– Middleware: same with Websphere
– Application teams: why not creating recipe to deploy application ?
• Talk to your boss to tell him:
– You’ll be faster doing your job with Chef
– Automation = avoiding human error
– You’ll need less people to do more …. And you’ll get more time to do
something else.
• Rome wasn’t build in a day ! It will takes time but it is worth it.
© Copyright IBM Corporation 2015
Thank you !
Special thanks to (for their support or help)
- Jay Kruemcke @chromeaix @cloudrancher
- Philippe Hermes @phhermes
- Nigel Griffiths @mr_nmon / Gareth Coates @power_gaz
- Chris Gibson @cgibbo
- Rosa Davidson @radavids1
- Alain Dejoux @adejoux  Alain is helping writing the providers | SPECIAL THANK YOU !
- Chef / Opscode @julian_dunn  Julian is product Manager at Chef
© Copyright IBM Corporation 2015
Continue growing your IBM skills
ibm.com/training provides a
comprehensive portfolio of skills and
career
accelerators that are designed to
meet all
your training needs.
• Training in cities local to you - where and
when you need it, and in the format you want
– Use IBM Training Search to locate public training classes
near to you with our five Global Training Providers
– Private training is also available with our Global Training Providers
• Demanding a high standard of quality –
view the paths to success
– Browse Training Paths and Certifications to find the
course that is right for you
• If you can’t find the training that is right for you with
our Global Training Providers, we can help.
– Contact IBM Training at dpmc@us.ibm.com
Global Skills
Initiative
45
© Copyright IBM Corporation
2015