Installing Chef Delivery, AWS This topic guides you through setting up Chef Delivery for AWS provisioning with the delivery-cluster cookbook and Chef provisioning, and validating the installation once Chef Delivery is set up. With the delivery-cluster cookbook, you install the Chef Delivery cluster on your own infrastructure using Chef provisioning and your license key. Process Overview The process for installing Chef Delivery using the delivery-cluster cookbook involves the following tasks: Create and configure a dedicated provisioning node Configure the delivery-cluster cookbook to use Amazon Web Services (AWS) provisioning Run automation for delivery-cluster, which sets up a full Chef Delivery cluster and gives you login credentials Verify the Chef Delivery provisioning Set up your workstation to develop projects in Chef Delivery Configure Chef Delivery Validate the installation by creating a cookbook and kicking off a pipeline Prerequisites Chef Delivery requires a license from Chef to install. Supported Platforms Chef Delivery may be run on the following platforms. Do not mix platform versions within the Chef Delivery cluster. OS Versions Ubuntu 12.04, 14.04 Centos 6.5, 6.6, 7 Redhat 6.5, 6.6, 7 Infrastructure Chef Delivery has the following infrastructure requirements: Function vCPU Free disk space (in /var) RAM Provisioning Node 1 (2.5GHz) 1GB 8GB Delivery Server 4 8GB 80GB Chef Server (must be v12). See additional information in note, below. 4 8GB 80GB Supermarket Server 4 8GB 80GB Build Nodes (at least 3, one for each Verify phase: Lint, Syntax, and Functional) 2 4GB 60GB Infrastructure Nodes (at least 4, one for each environment stage: Acceptance, Union, Rehearsal, and Delivered) 2 (for test purposes) 4GB (for test purposes) 60GB (for test purposes) NOTE: If you want to use your existing Chef server in the Chef Delivery cluster, the version of the Chef server must be 12.2, it cannot have a high availability configuration. Network and Ports Chef Delivery has the following network and port requirements: Ports Description Server State 10000-10003 TCP Push Jobs Chef server LISTEN 8989 TCP Delivery Git (SCM) Delivery server LISTEN 443 TCP HTTP Secure Chef server, Delivery server LISTEN 22 TCP SSH All 80 TCP HTTP Chef server, Delivery server LISTEN 5672 TCP Analytics MQ n/a n/a 10012-10013 TCP Analytics Messages/Notifier n/a n/a LISTEN Install Chef Delivery Begin installing Chef Delivery by creating a provisioning node to provision and maintain the cluster. Provisioning Node Chef Delivery uses Chef provisioning to create, upgrade, and manage the cluster. A dedicated provisioning node ensures a central place from which the Chef Delivery cluster is managed, including managing those allowed to control it. Use your existing means to provision a dedicated provisioning node with a supported platform. This machine does not need to be powerful because it runs only provisioning code. Within Amazon Web Services (AWS), Chef generally uses a t2.micro instance, which is a single CPU 2.5GHz, 1 GB of memory, and 8 GB of disk space. On the provisioning node: 1. Install a development environment: For Debian-based (apt): $ sudo apt-get install build-essential For Red Hat Enterprise Linux-based (yum): $ sudo yum groupinstall "Development Tools" For Mac OS X: $ sudo xcode-select --install 2. Install Git and configure your GitHub username and email, for help see: Installing Git and First-Time Git Setup. 3. Install the Chef Development Kit. Be sure to set the system Ruby; for details, see Add Ruby to $PATH. 4. Install the Chef Delivery knife push utility: $ chef gem install knife-push 5. Move your Chef Delivery license key to ~/delivery.license. 6. Clone the Git repo for the Chef Delivery cluster: $ git clone https://github.com/opscode-cookbooks/delivery-cluster.git ~/delivery-cluster AWS Settings The following section details settings for Chef Delivery with AWS provisioning. Option Description Environment Name This name becomes the default name for your Chef Delivery cluster environment, enterprise, and organization. The naming convention prohibits the use of capital letters, but should otherwise be alphanumeric. Key Name This is the name of your Amazon Web Services (AWS) .pem credential file. Option Description Enter only the name, without the extension. This file is typically located in the ~.ssh directory, and is associated with the Key name pair value in Amazon Web Services (AWS) console: https://console.aws.amazon.com/iam/home. Image ID, Subnet ID, and Security Group ID Use existing chef-server This is the information that Amazon Web Services (AWS) needs to provision the nodes. It is wise to verify that the given defaults are actually the ones that you want by opening your Amazon Web Services (AWS) console and checking. Accept the default, “no”, if you want delivery-cluster to configure a Chef server for you (you will be asked for the IP address of the server). Enter “yes” if you have a configured Chef server that you want to use and do not need delivery-cluster to configure one for you. If you do specify “yes”, please note that the version of the Chef server must be 12.2, it cannot have a high availability configuration. In addition, a special entry must be made to the environments configuration file (environments/NAME_OF_ENV.json) similar to: "chef-server": { "organization": "test", "existing": true, "host": "chef-server.example.com" }, License File The path to your Chef Delivery license file; the default path is your home directory. Enable Supermarket Enter “yes” if you intend to run cookbooks through your Chef Delivery pipeline. AWS Provisioning This section describes setting up a Chef Delivery cluster to use AWS for provisioning. This process creates at least five machines, depending on the selected configuration options. The typical (and recommended) starting setup is five nodes in addition to the provisioning node: One to run the Chef server One to run Chef Delivery Three to act as build nodes One to run the Chef Supermarket server; for cookbook dependency resolution, using Chef Supermarket server is required For a test environment, use stand-in infrastructure nodes; we recommend four, one for each environment stage: Acceptance, Union, Rehearsal, and Delivered On the provisioning node: 1. Configure your Amazon Web Services (AWS) key credentials in two files, ~/.aws/config and ~/.aws/credentials as shown, with the text editor of your choice: For ~/.aws/config: [default] region = us-west-2 output = text aws_access_key_id = YOUR_ACCESS_KEY_ID aws_secret_access_key = YOUR_SECRET_KEY For ~/.aws/credentials: [default] aws_access_key_id = YOUR_ACCESS_KEY_ID aws_secret_access_key = YOUR_SECRET_KEY 2. Transfer the private key of the key pair (private-key) that you will use to create the servers in Amazon Web Services (AWS), to ~/.ssh on the provisioning node. If you do not have one already, use this link and create one: Amazon EC2 Key Pairs. 3. Ensure that an Amazon Web Services (AWS) security group is available, with the appropriate ports open. See Network and Ports for details. 4. Move to the delivery-cluster directory: cd ~/delivery-cluster. 5. Create a configuration file to drive Chef provisioning: $ rake setup:generate_env This opens a wizard, the following code block shows the interactions. Variables you must enter are shown in ALL CAPs. Default values are shown in square brackets; pressing ENT sets the default value. This command creates an environments/ENV_NAME.json file with the specified settings that you can modify if need be. $ rake setup:generate_env Gathering Cluster Information Provide the following information to generate your environment. Global Attributes Environment Name [test]: Cluster ID [test]: Available Drivers: [ aws | ssh | vagrant ] Driver Name [vagrant]: aws Driver Information [aws] Key Name: MY-KEY-NAME SSH Username [ubuntu]: Image ID [ami-3d50120d]: Subnet ID [subnet-19ac017c]: Security Group ID [sg-cbacf8ae]: Use private ip for ssh? [yes]: Would you like to configure Proxy Settings? [no]: Chef Server Organization Name [test]: Use existing chef-server? [no]: Flavor [c3.xlarge]: Delivery Server Package Version [latest]: Enterprise Name [test]: License File [/Users/YOUR-NAME/delivery.license]: Flavor [c3.xlarge]: Analytics Server Enable Analytics? [no]: Supermarket Server Enable Supermarket? [no]: yes Flavor [c3.xlarge]: Build Nodes Number of Build Nodes [1]: 3 Flavor [c3.large]: Specify a delivery-cli artifact? [no]: Rendering Environment => environments/test.json 6. Important! Export your new environment by executing the following: (example uses test as the environment name): $ export CHEF_ENV=test NOTE: You can name your environment file anything, but note the name for future reference. Run Provisioning To run provisioning issue this command from inside the delivery-cluster directory: $ rake setup:cluster NOTES: If the first converge fails on the build nodes, try running the above step again. The Chef Delivery cluster is complicated and sometimes there are timeouts before the setup completes. Running provisioning creates the requested instances on Amazon Web Services (AWS). If there are any failures, check the Amazon Web Services (AWS) console for nodes without names. Nodes without names can be removed. Verify Provisioning To verify provisioning do the following, from inside the delivery-cluster directory: 1. Run this command: $ rake info:list_core_services 2. Navigate to your CHEF_SERVER_URL, and login with delivery:delivery. 3. Click on Nodes. You’ll see your Chef Delivery server, your build nodes, and nodes for any other optional features that you installed. 4. Get the credentials and URL with this command: $ rake info:delivery_creds 5. Navigate to the Web Login and use the admin credentials to log in. 6. Run this command: $ knife node status All build nodes should report available. Configure Delivery In Chef Delivery there are multiple levels of organization: enterprises, organizations, and projects. The provisioning step created the initial enterprise you specified in your environment file as the first name option. Enterprises are designed to provide units of multi-tenancy with separate sets of organizations and users. Begin configuring Chef Delivery by adding users and organizations. NOTE: Chef Delivery, by default, hosts a Git server that you interact with through the delivery CLI commands. Additionally, you can integrate GitHub Enterprise or GitHub.com. Add Users Chef recommends that the default admin account not be used after Chef Delivery is installed. Instead, use the following web UI procedure to create a new user for yourself. Then log out as admin and log back in as the user you created. 1. Log into the Chef Delivery web UI as an administrator. 2. Select Users from the drop-down menu on the upper right. The Users list page opens. You can use the search filter in the upper right corner to make sure that the user is not already added. 3. Click the plus sign (+) next to Add a New User. 4. In the Add a New User text area, select Internal, which means you are manually adding the user to the Chef Delivery database. The selection box is grey for the active selection. Options for Name and Email, first name, last name, email address, and Security Information, a login name and password, appear. 5. Enter the appropriate information. Select Admin in the Roles Within the Enterprise option list. Click Save and Close. The User list page opens and a status message appears. To check, click Edit and verify that the user details are present. Create Organizations Chef suggests creating a sandbox organization in which a test project can be run for verification and testing purposes. Additionally, it is recommended that each organization created corresponds to a working directory within which projects for that organization are managed. To create an organization, do the following: 1. Log into the Chef Delivery web UI as the Admin user you created; use this command (run in your delivery-cluster directory) to find the credentials: $ rake info:delivery_creds 2. Select Organizations from the drop-down menu on the upper right. The organizations list page opens. 3. Click the plus sign (+) next to Add a New Organization. 4. Enter an organization name in the text area, and then click Save & Close. Setup a Workstation After the Chef Delivery cluster has been created, team members need to install the Chef Delivery CLI on their workstations. NOTE: If you reuse the provisioning node in the previous section of this installation guide as a workstation for using Chef Delivery, you can skip steps 1 through 4 below. Setting up a workstation requires that you do the following: 1. Install a development environment: Debian based (apt): apt-get install build-essential RHEL based (yum): yum groupinstall "Development Tools" OS X: xcode-select --install 2. Install Git: http://git-scm.com/book/en/v2/Getting-Started-Installing-Git. Also, make sure that your SSH public key is added to your user profile in Chef Delivery. 3. Install the Chef Development Kit: https://downloads.chef.io/chef-dk/. Be sure to follow all instructions; for example, to run delivery commands you must set the following: o o o echo 'eval "$(chef shell-init bash)"' >> ~/.bash_profile 4. Install the Chef knife push utility: chef gem install knife-push. 5. Install and configure the Chef Delivery CLI, described next. Install the Chef Delivery CLI The delivery command line interface (CLI) is required for the workstation and for many Chef Delivery functions. Mac OS X For Mac OS X, run the following commands: 1. Download the package https://s3.amazonaws.com/delivery-packages/cli/deliverycli0.0.0%2B20151118205039-1.pkg. 2. Click the package and install. 3. You may need to allow the package on the machine. To do this, open System Preferences -> Security & Privacy, and click Allow. You can also go to your Downloads folder and right-click the package to open the installer; simply accept the agreement and the command line interface is installed. Ubuntu For Ubuntu 14.04, run the following commands: $ curl https://packagecloud.io/install/repositories/chef/current/script.deb.sh | sudo bash $ sudo apt-get install delivery-cli RHEL, SUSE For Red Hat Enterprise Linux 6.5 and SUSE Enterprise Linux Server 6.5, run the following commands: $ curl https://packagecloud.io/install/repositories/chef/current/script.rpm.sh | sudo bash $ sudo yum install delivery-cli.rpm Microsoft Windows For Microsoft Windows: 1. Download the binary from https://s3.amazonaws.com/delivery-packages/cli/deliverycli-0.0.0%2B20151020165859-1-x64.msi. 2. Install Git (2.5) on Microsoft Windows: https://github.com/git-forwindows/git/releases/download/v2.5.3.windows.1/Git-2.5.3-64-bit.exe. WARNING: Git 2.6 won’t work. 3. Modify the PATH environment variable to the path to the location of the delivery command line interface. For example: C:\\chef\\delivery-cli\\bin Configure the Chef Delivery CLI Before you use the Chef Delivery CLI from a workstation, you need to provide it with details such as the URL of the Delivery server, and the names of the relevant enterprise, organization, and user. The delivery setup subcommand creates a configuration file named .delivery/cli.toml with the required information. The placement of the .delivery directory in your file hierarchy is significant. Like Git, Chef Delivery CLI commands search the current directory and parent directories to locate server settings. Because server settings are unique to an organization, we recommend that you create a directory for each organization as required and run the delivery setup command from each of those directories. $ delivery setup --server=DELIVERY_SERVER_IP_ADDR --ent=ENTERPRISE -org=ORGANIZATION --user=USERNAME Authenticate You must authenticate access to the Chef Delivery Git server, which runs on port 8989 by default. NOTE: This authentication step is for the Git server that comes with Chef Delivery. If you actually integrate with GitHub, you do not need to do this. Run the following SSH command; enter “Yes” when prompted. $ ssh -l USER@ENTERPRISE -p 8989 DELIVERY_SERVER_IP_ADDRESS You should get output similar to this: $ ssh -l sally@chef -p 8989 10.194.11.99 The authenticity of host '[10.194.11.99]:8989 ([10.194.11.99]:8989)' can't be established. RSA key fingerprint is ba:db:0c:97:f8:d4:6d:0f:0b:57:0d:0f:0e:a4:15:01. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '[10.194.11.99]:8989' (RSA) to the list of known hosts. channel 0: protocol error: close rcvd twice Hi sally@chef! You've successfully authenticated, but Chef Delivery does not provide shell access. Connection to 10.194.11.99 closed. Your workstation is now set up for use with Chef Delivery. Validate the Installation The surest way to validate your Chef Delivery installation is by creating a cookbook and submitting it to GitHub, which kicks off a pipeline allowing you to see how the process works. If a project is a cookbook, we recommend starting with delivery-truck, an open source build cookbook created for driving cookbook pipelines in Chef Delivery. You can customize some aspects of delivery-truck through your project’s .delivery/config.json. To have more control, or to opt-out of some of the behavior of delivery-truck, create a wrapper build cookbook. To learn more about wrapper cookbooks, see Chef Supermarket -> Wrapper Cookbooks and the blog post Doing Wrapper Cookbooks Right. NOTE: These instructions assume that you will use Chef Delivery as your source code source of truth and that Chef Delivery is not integrated with GitHub Enterprise or GitHub.com. This topic describes the recommended setup for a Chef cookbook project using Chef Delivery. The following example shows how to create a cookbook, with project and pipeline, configure it to be built with delivery-truck, and then import the new cookbook into Chef Delivery. From your workstation, do the following: 1. Make a working directory (workspace in the example): $ mkdir ~/workspace && cd ~/workspace 2. Setup the delivery-cli to, by default, contact the appropriate delivery server at SERVER, with a default ENTERPRISE and ORGANIZATION: $ delivery setup --server=SERVER --ent=ENTERPRISE --org=ORGANIZATION -user=USERNAME NOTE: The server, enterprise, organization, and user must already exist. 3. Create a cookbook: $ chef generate cookbook NEW-COOKBOOK-NAME $ cd NEW-COOKBOOK-NAME This uses the Chef Development Kit to generate a new cookbook, including a default recipe and default ChefSpec tests. 4. Create an initial commit on the master branch (use git status to verify the change): $ git add . $ git commit -m 'Initial Commit' Running chef generate initialized a Git repository automatically for this cookbook. If you created the build cookbook manually, initialize the Git repository with the git init command. 5. Get an API token for Chef Delivery: $ delivery token 6. Initialize the cookbook for Chef Delivery: $ delivery init This creates a new project in Chef Delivery, pushes the master branch, creates a feature branch, generates a default Chef Delivery project configuration file, pushes the first change for review, and then opens a browser window that shows the change.