DevOps
Week 1
Eoin Rogers
eoin.rogers@tudublin.ie
The traditional way of doing things
The DevOps way of doing things
Minimise the distinction between these
roles.
People doing the development and the
operations work share and collaborate
extensively.
But note: generally not trying to eliminate
these distinctions!
Definition
DevOps is the practice of operations and
development engineers participating together in
the entire service lifecycle, from design through
the development process to production support
From https://theagileadmin.com/what-is-devops/
CAMS – Culture, Automation, Measurement,
Sharing
Culture
DevOps implies a fundamental
change in the way that a company
or organisation works, with
extensive use of scrum and
collaboration
Automation
The setup and maintenance of both
development and operations platforms should be
made as automated as possible
Measurement
Everything that can be measured and logged
should be. Measurments are used to guide
decisions: do what is shown to work, not what
you think will work
Sharing
Everyone involved should have access to the
code and infrastructure
It’s OK to be sceptical!
●
This is all just a load of buzzwords, isn’t it?
–
Maybe, but the technologies used are very useful,
and already seeing wide use
–
Even if you don’t use this stuff, it will help you
communicate with others
What things will I have to know before I take the
course?
Not much!
●
A working knowledge of at least one mainstream programming
language
–
●
Some degree of proficiency with a Linux/Unix command line
–
●
Could be Java, Python, C#, C/C++, whatever
Not expecting some sort of genius, but know how to use things like cd, ls,
touch etc.
You have some idea (however limited!) of the types of things that
are needed to maintain a large, complex computer/software system
–
Writing code, dealing with bug reports, liaising with users, deploying and
maintaining servers, etc.
What are the specific technologies this course will
cover?
Shell scripting
Version control
Virtualisation
Virtualisation
Cloud computing
Containerisation
Infrastructure as code
Continuous integration
Assessment
●
60% exam
●
40% continuous assessment
–
●
Labs
First lab is on Friday at 11am
Introduction to DevOps
Lecture 2
Shell scripting
Eoin Rogers
eoin.rogers@tudublin.ie
The traditional way of doing things
The DevOps way of doing things
Minimise the distinction between these
roles.
People doing the development and the
operations work share and collaborate
extensively.
But note: generally not trying to eliminate
these distinctions!
Bash
Some common commands
●
ls
●
pwd
●
time
●
cal
●
mkdir
●
rmdir
Some common commands
●
ls List contents of the current directory
●
pwd
●
time
●
cal
●
mkdir
●
rmdir
Some common commands
●
ls List contents of the current directory
●
pwd Show the present working directory
●
time
●
cal
●
mkdir
●
rmdir
Some common commands
●
ls List contents of the current directory
●
pwd Show the present working directory
●
time Show the current date and time
●
cal
●
mkdir
●
rmdir
Some common commands
●
ls List contents of the current directory
●
pwd Show the present working directory
●
time Show the current date and time
●
cal Show a calendar with today’s date
●
mkdir
●
rmdir
Some common commands
●
ls List contents of the current directory
●
pwd Show the present working directory
●
time Show the current date and time
●
cal Show a calendar with today’s date
●
mkdir Create a new directory
●
rmdir
Some common commands
●
ls List contents of the current directory
●
pwd Show the present working directory
●
time Show the current date and time
●
cal Show a calendar with today’s date
●
mkdir Create a new directory
●
rmdir Remove a directory
Some common commands
●
cp
●
mv
●
grep
●
find
●
touch
●
rm
●
wc
●
man
Some common commands
●
cp Copy file
●
mv
●
grep
●
find
●
touch
●
rm
●
wc
●
man
Some common commands
●
cp Copy file
●
mv Move file
●
grep
●
find
●
touch
●
rm
●
wc
●
man
Some common commands
●
cp Copy file
●
mv Move file
●
grep Search file for string
●
find
●
touch
●
rm
●
wc
●
man
Some common commands
●
cp Copy file
●
mv Move file
●
grep Search file for string
●
find Search a file by name, type or contents
●
touch
●
rm
●
wc
●
man
Some common commands
●
cp Copy file
●
mv Move file
●
grep Search file for string
●
find Search a file by name, type or contents
●
touch Update the timestamp of a file (or create it, if it doesn’t exist)
●
rm
●
wc
●
man
Some common commands
●
cp Copy file
●
mv Move file
●
grep Search file for string
●
find Search a file by name, type or contents
●
touch Update the timestamp of a file (or create it, if it doesn’t exist)
●
rm Remove a file
●
wc
●
man
Some common commands
●
cp Copy file
●
mv Move file
●
grep Search file for string
●
find Search a file by name, type or contents
●
touch Update the timestamp of a file (or create it, if it doesn’t exist)
●
rm Remove a file
●
wc Word count – count the number of words or lines in a file
●
man
Some common commands
●
cp Copy file
●
mv Move file
●
grep Search file for string
●
find Search a file by name, type or contents
●
touch Update the timestamp of a file (or create it, if it doesn’t exist)
●
rm Remove a file
●
wc Word count – count the number of words or lines in a file
●
man Display the manual for a command
Relative and absoulte path
●
Relative path (directory called usr in current
directory)
–
●
usr/example
Absolute path (directory called usr at root of the
filesystem)
–
/usr/example
Current and parent directory
●
Current directory: .
●
Parent directory: ..
●
Grandparent directory: ../..
●
Great-grandparent directory: ../../..
●
etc...
Other directories
●
Root directory: /
●
Home directory
●
–
/home/<user name>
–
~
Most programs are in /usr
Putting commands into a script
#! /bin/bash
ls
Add execution permissions
chmod +x my_script.sh
Echo (print statement)
echo “Hello, world!”
Variables
location=”world”
echo “Hello, $location!”
Variables can hold paths
location=/bin/bash
echo “Running from $location”
...or the output from commands
name=$(whoami)
echo “Hello, $name!”
Copy your home folder
#!/bin/bash
name=$(whoami)
old=/home/$name
new=/home/${name}_backup
cp $old/* $new
Redirecting output
some command > example.txt
some command 2> example.txt
some command &> example.txt
Redirecting input
some command < example.txt
some command <<< "Some text"
some command <<terminating
Some text,
Spread over multiple lines!
terminating
Pipes
command 1 | command 2
Useful examples
Show all PDF files in the current directory:
ls *.pdf
Useful examples
Show every file beginning with the letter X on your
computer:
find / -iname "x*" 2> /dev/null
Useful examples
Show which version of the Linux kernel you are
using:
uname -a
Useful examples
Delete everything in directory dir, and then delete
the directory itself:
rmdir -rf
Useful examples
Find all instances of the string “hello” in the file
f.txt:
grep hello f.txt
Useful examples
Find all instances of the string “hello” in the any
file in the directory example:
grep -rn hello example
Maths expressions
expr 1 + 1
expr 10 – 5
expr 8 \* 2
Arithmetic expansion
x=$((1 + 1))
y=$(( (8 / 2) * 15 ))
Let
let x=(1+1)*5
let x++
let x+=10
Boolean expressions
[ “one” = “two” ]
echo $?
[ “one” = “one” ]
echo $?
Boolean expressions
[ 1 -eq 2 ]
echo $?
[ 1 -eq 1 ]
echo $?
Boolean operators
●
For strings:
–
●
=, !=, >, <
For numbers
–
-eq, -ne, -gt, -lt, -ge, -le
Boolean expressions
[ “one” = “one” ] && [ 1 -eq 1 ]
Boolean expressions
(test "one" = "one") && (test 1 -eq 1)
If statement
#!/bin/bash
one=100
two=200
if [ $one -lt $two ]; then
echo "$one is less than $two"
fi
#!/bin/bash
If-else statement
one=100
two=200
if [ $one -lt $two ]; then
echo "$one is less than $two"
else
echo "$one is not less than $two"
fi
If-elif-else statement
#!/bin/bash
one=100
two=200
if [ $one -lt $two ]; then
echo "$one is less than $two"
elif [ $one -eq $two ]; then
echo "$one is equal to $two"
else
echo "$one is not less than $two"
fi
Case statement (like switch)
#!/bin/bash
echo "Enter a number: "
read number
case $(( $number % 2)) in
0)
num_type="even"
;;
1)
num_type="odd"
;;
esac
printf "%d is an %s number\n" $number $num_type
#!/bin/bash
Loops
count_to=100
v=1
while [ $v -le $count_to ]; do
echo $v
let v++
done
#!/bin/bash
Loops
count_to=100
v=1
until [ $v -gt $count_to ]; do
echo $v
let v++
done
Parameters
#!/bin/bash
echo "You passed $1 to this script"
Parameters
./my_script.sh hello
You passed hello to this script
#!/bin/bash
Functions
count_files() {
ls -l $1 | wc -l
}
copy_files() {
cp $1/* $2
}
num_files=$(count_files $1)
echo "Copying $num_files in $1 to $2"
copy_files $1 $2
And that’s all for today!
:-)
Introduction to DevOps
Lecture 3
Version control
Eoin Rogers
eoin.rogers@tudublin.ie
What did CAMS stand for?
CAMS
What did CAMS stand for?
CAMS
Sharing everything with everyone
What is version control?
Git
Alternatives to git
●
Mercurial
●
Fossil
●
BitKeeper
●
Bazaar
●
Subversion (SVN)
What is a git repo?
●
●
A directory (folder) containing the files and other
directories you want git to track
Contains a sub-folder called .git, which is where
git stores the stuff it uses internally (including
the commit history)
Obtaining a git repo
●
●
git init – Make a new repo from a directory
git clone <URL> – Make a copy of an existing
repo (usually hosted on an external server)
Moving around
●
git checkout <hex code> – Switch to a previous
commit
Tagging
●
●
●
Tags are commits that you want to give a humanreadable name to (for example, to identify particular
releases of the program)
git tag <name> – Create a new tag at the current
commit
The checkout command also accepts tag names, as
well as the usual hex IDs!
Tagging
Adding and committing
●
●
git add <file> -- Add a file or folder to the repo
git rm <file> -- Remove a file or folder from the
repo
●
git commit – Commit the changes to the repo
●
We usually call commit with a message after it:
git commit -m "Changed the colour of the login screen"
Git uses a staging area
View the internal status of git
git status
Branches
●
●
A branch is a parallel development history which split
off from the main repo
This is usually used to allow developers to work on
features without interfering with the main development
effort.
●
git branch <name> -- create a new branch
●
We can use checkout to jump to another branch
Branches
Merging and rebasing
●
●
●
●
Sometimes, we might want to bring changes in one branch into
another branch
git merge <branch> -- Merge the current branch with the branch
passed to the command
If merge conflicts occur, git will tell you, and ask you to resolve them
before it continues
git rebase <branch> -- Apply the changes from the branch passed to
the command to the current branch, thus changing the current
branche’s history
Pushing code
●
●
●
Git has a concept of remotes – similar to tags, but
refer to a copy of the repo on a remote server
When you change your code, you’ll generally want to
send the changes back to the remote. This is called
pushing.
git push <remote> <branch> -- Send a branch to a
remote
Avoiding the CLI
●
●
Lots of GUI tools for git exist
–
Some are self-contained applications
–
Lots are plugins for existing IDEs and editors
–
For example, git integration is available for Eclipse, VSCode etc.
But we’re going to stick to the CLI in this course
–
It’s good to understand the CLI stuff (the GUIs generally can’t do
everything)
–
Avoid installing GUI applications on our VMs, since they are quite
resource-heavy
Remove untracked files
git clean
N.B. This cannot be
undone!!!
Resources to learn git
git-scm.com/book
Thanks for your attention!
Introduction to DevOps
Lecture 3
The DevOps environment
Eoin Rogers
eoin.rogers@tudublin.ie
What we’ve done so far...
●
The basic ‘idea’ of DevOps
–
●
Making development and operations people get
along better
Key technologies
–
Shell scripting
–
Version control
What we’ve done so far...
Golden images
We can’t just concentrate on technologies for
this course...
(or at least we shouldn’t)
Think back to CAMS
●
Culture
●
Automation
●
Measurment
●
Sharing
Think back to CAMS
●
Culture
●
Automation
●
Measurment
●
Sharing
In effect, culture must
preceed everything else...
These people all have different
perspectives...
●
Developers
●
Operations
●
Managers
●
Security
●
Users
Developers
●
Write code
●
Implement features
●
Fix bugs
●
Always looking to the next big thing
Developer incentives
●
Looking to make big launches
–
●
●
See big launches as accomplishments
Often willing to change things, learn new
technologies etc.
Risk takers
Operations
●
Run services and hardware
●
Administrators
●
Troubleshooting
●
Monitoring
●
Incident management
●
Their job is more ongoing, and less deadline focused than
the developer’s
Operations incentives
●
●
Far more conservative than developers
–
Don’t like taking risks
–
Don’t like big changes
–
Like to opt for a slow and steady approach
Often invisible within an organisation
–
Often (unfairly!) blamed when things go wrong
See the problem yet?
There’s a big clash of cultures going on here...
Plus there’s more!
Management
●
Highly feature oriented
●
Probably interact directly with the customer
–
●
●
Liable to make promises that the dev and ops people can’t
really be expected to keep…
Often won’t even interact directly with the ops team
Will be worried about non-technical issues like the
budget as well
Security
●
●
Even more change-averse than the ops people
Don’t really care about uptime and user
experience – they just want to make sure that
the system is and remains secure!
Conflict!
Conflict!
Conflict!
Conflict!
Conflict!
DevOps is an attempt to address this conflict
But how?
●
●
●
●
●
Everyone works on a shared environment (VM?)
Some way to modify the environment and push those
changes to everyone
Agile development for both the codebase the developers
and working on and the shared environment
Small, incremental changes, no more big changes that
break everything
All technologies must be approved by all stakeholders,
one group can’t just go off and work on their own...
But how?
●
●
●
●
●
Everyone works on a shared environment (VM?)
Some way to modify the environment and push those
changes to everyone
Agile development for both the codebase the developers
and working on and the shared environment
Small, incremental changes, no more big changes that
break everything
All technologies must be approved by all stakeholders,
one group can’t just go off and work on their own...
Two main techniques
●
Golden image
–
●
Config management
–
●
Shared VM between everyone
Write shell scripts to automate setting up
everyone’s machine the same way
Can use a hybrid of these two approaches
This course will mostly concentrate on golden
images
Packer
●
●
●
Developed by HashiCorp, a major vendor of
DevOps and automation tools
Can create and configure VMs and cloud
services
Useful for creating golden images
Packer
●
Packer reads in config files called templates,
which are written in JSON
●
How much does everybody know about JSON?
●
Let’s review just in case it isn’t much!
JavaScript Object Notation (JSON)
{ "name" : "eoin", "job" : "lecturer", "employer" :
"TU Dublin" }
JSON types
Strings
"Hello, World!"
JSON types
Integers
0, 1, -2, 128, -1234
JSON types
Floating-point values
3.14, 2.718, 1.23
JSON types
Booleans
true, false
JSON types
Arrays
[true, "Hello, World!", 123]
JSON types
Null values
null
JSON types
Objects
{ "example" : [true, "Hello, World!", 123] }
Canonical Packer template
{
"builders" : [],
"description" : "...",
"min_packer_version" : "...",
"post-processors" : [],
"provisioners" : [],
"variables" : {}
}
Canonical Packer template
{
"builders" : [],
"description" : "...",
"min_packer_version" : "...",
"post-processors" : [],
"provisioners" : [],
"variables" : {}
}
Mandatory!
Builders
●
Things in Packer which create VMs/machines
–
Packer comes with a selection of builtin builders for
common VM/cloud platforms
●
●
●
–
VirtualBox
AWS
Etc.
Can add new builders for other platforms
{
VirtualBox builder
"type": "virtualbox-iso",
"guest_os_type": "Ubuntu_64",
"iso_url":
"http://releases.ubuntu.com/12.04/ubuntu-12.04.5server-amd64.iso",
"iso_checksum":
"md5:769474248a3897f4865817446f9a4a53",
"ssh_username": "packer",
"ssh_password": "packer",
"shutdown_command": "echo 'packer' | sudo -S
shutdown -P now"
}
Vagrant builder
"builders": [
{
"communicator": "ssh",
"source_path": "hashicorp/precise64",
"provider": "virtualbox",
"add_force": true,
"type": "vagrant"
}
]
Provisioners
{
}
"type": "shell",
"script": "script.sh"
Variables
"variables": {
"aws_access_key": "",
"aws_secret_key": ""
}
Post-processors
"post-processors": [
{
"type": "compress",
"format": "tar.gz"
}
]
HashiCorp Config Language (HCL)
{
"example" : {
"number" : 1.2,
"string" : "Hello, World!"
}
}
HashiCorp Config Language (HCL)
example {
number = 1.2
string = "Hello, World!"
}
Let’s move on to talking about Vagrant
Vagrant intialisation
vagrant init hashicorp/bionic64
Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = "hashicorp/bionic64"
end
Start the box (VM)
vagrant up
SSH into the box
vagrant ssh
A more complex Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = "hashicorp/bionic64"
config.vm.provision :shell, path: "bootstrap.sh"
config.vm.network :forwarded_port, guest: 80,
host: 4567
end
Deleting the Vagrant box
vagrant destroy
vagrant box remove hashicorp/bionic64
That’s all for this week!
Thanks for your attention
Introduction to DevOps
Lecture 5
Build Systems & Continuous integration
Eoin Rogers
eoin.rogers@tudublin.ie
Build systems
●
●
●
If using a compiled language, going from source code to a
executable binary typically involves compiling and linking
This could involve a lot of work! For example, Chromium
(the open-source version of Google Chrome) consists of
about 35 million lines of code spread across 56,000 C++
files!
Typing in 56,000 compile commands would be absurd: we
need to find some way to automate this…
Build systems
●
●
●
We could use a script, but that script would be large
and hard to maintain.
A script won’t give us incremental compilation (only
re-compile files that have changed since the last
compilation run!)
For this reason, we generally use dedicated build
systems.
Common build systems
●
GNU make, Ninja
●
CMake
●
Maven, Gradle
●
Bazel, Buck, Please
●
SCons, Meson
GNU make
●
Commonly used for C and C++
●
Quite low-level, and can be slow
●
However, it is very easy to use!
●
To use, put a file called Makefile in the root of
the repository
Dependency graphs
Makefile
Basic Makefile syntax
output: input
command(s)
N.B. The indentation must be a tab!
Let the user choose a C compiler
Variables, wildcards and functions
Phony targets
General Makefile tips
●
●
●
By default, make will build the first target that
doesn’t start with a dot.
You can override this by specifying a target
(e.g. make clean)
Script code outside of targets (variables, etc.)
will always run.
Continuous integration
CI systems are build systems on steroids: build,
test integrate and deploy!
Where does this fit in CAMS?
CAMS
Where does this fit in CAMS?
CAMS
Technological enablers for
continuous integration
●
●
●
Have a monorepo, put all of the code there
Everyone contributes towards the codebase and
environment
Every so often (maybe every night, maybe every hour), some
automated system kicks in and:
–
Builds environment and software
–
Deploys/stages the code to the environment/VM
–
Runs tests on the code
–
Deploys the code to the production environment if the tests pass
(and alerts someone if they don’t!)
Common CI tools
●
Jenkins
●
Bamboo
●
Buildbot
●
GitLab
●
Travis
Jenkins
Jenkins
●
Dedicated continuous integration tool
–
●
●
●
●
Is that useful?
Written in Java
Intended to be run on a remote server, provides a web-based
interface
Can run as a distributed system, with distinct controllers (which
send tasks to agents) and agents (which actually run builds)
Controlled using a script called a Jenkinsfile
Jenkins Pipeline
●
●
A Jenkinsfile specifies a set of tasks called a pipeline
Typically, this will implement the steps of a full
continuous integration cycle
–
Build
–
Test
–
Integrate deploy
Jenkinsfile
pipeline {
agent any
}
// Stages go here
Jenkinsfile
stages {
stage('Build program') {
steps {
sh 'make'
}
}
}
stage('Test program') {
steps {
sh 'make run_tests'
}
}
Jenkins web interface
That’s all for this week!
Thanks for your attention
Introduction to DevOps
Lecture 6
Containers and Docker
Eoin Rogers
eoin.rogers@tudublin.ie
Virtualisation vs. Emulation
●
●
Emulation – Simulating the behaviour of a
computer using software (i.e. software
implementation of hardware)
Virtualisation – Like emulation, but with
hardware acceleration (which makes it much
faster!)
Imagine you’re running a web app...
Web app on a VM
Containerise your application
Some terminology
●
●
An image is a container on disk which isn’t
running
So the difference between a container and an
image is a bit like the difference between a
process and a program
Docker commands
●
docker build makes an image
●
docker run launches a built image
●
docker stop shuts down a container
●
docker rm deletes an image
To create a container image, we use a script
called a Dockerfile
Dockerfile command
COMMAND args
Dockerfile command
COMMAND args
Not case sensitive, but
conventionally written in uppercase
Some Dockerfile commands...
Derive one Dockerfile from another...
FROM ubuntu:22.04
Image
Tag
Some Dockerfile commands...
Run a command on a container during buildtime
RUN sudo apt install jenkins
Some Dockerfile commands...
Set an environment variable in the image
ENV VAR="this is a variable"
Some Dockerfile commands...
Set a variable within the Dockerfile
ARG VAR="this is a variable"
Some Dockerfile commands...
Require the user to specify a value when building
the Dockerfile
ARG VAR
Some Dockerfile commands...
Copy a file from the host to the image
COPY my_file.txt files/
Some Dockerfile commands...
Tell Docker what command must be run when the
container starts up
CMD my_app
Some Dockerfile commands...
Add metadata to an image
LABEL version="1.0"
Some Dockerfile commands...
Open a port on the container
EXPOSE 8080/tcp
Some Dockerfile commands...
Create and cd to a directory to work in
WORKDIR hello/world
That’s all for this week!
Thanks for your attention!
Introduction to DevOps
Lecture 8
Monitoring
Eoin Rogers
eoin.rogers@tudublin.ie
CAMS
CAMS
CAMS
Measurement
What are some reasons we would want to
measure our systems?
Empirical evidence
Predict future growth
Debugging
A/B testing
How do we collect information?
●
●
●
The application can write out logging information
We can monitor information provided by the wider
environment
–
Server logs
–
System/resource usage
–
VMs/containers can also log information (if we’re using
one)
External querying/probing of our system
What do we do with the information once
we’ve gathered it?
Prevent information overload
Provide summaries of data
Average response time: 2ms
Average CPU capacity for a single response: 40%
Average memory usage: 3%
Graphs and charts
Track how a statistic
changes over time
Software tools to visualise data
Nagios
Zabbix
Prometheus
Graphite
Grafana
That’s all for this week!
Thanks for your attention!
Introduction to DevOps
Lecture 9
Architecture
Eoin Rogers
eoin.rogers@tudublin.ie
Two types of architecture
●
Application architecture
vs
●
System architecture
We’ll mostly concentrate on the second kind.
If you’re interested in the first kind
The Architecture of Open Source Applications
http://aosabook.org
Some common system architectures
●
Client-server model/architecture
●
Peer-to-peer (not typical of a DevOps setup)
●
Model-View-Controller (MVC)
●
Data warehouse
●
Service-oriented architecture (SOA)
●
Microservices
●
Serverless computing
Client-server model
Peer-to-peer
MVC architecture
Data warehouse
Service-oriented architecture
Microservices
Serverless computing
Serverless is actually a
misnomer.
There’s still a server, but it is ephemeral
and stateless.
This is a type of event-driven
programming.
That’s all for this week!
Thanks for your attention!
Introduction to DevOps
Lecture 10
Software testing & profiling
Eoin Rogers
eoin.rogers@tudublin.ie
Program testing can be used to show the
presence of bugs, but never to show their
absence!
– Edsger Dijkstra
Testing lifecycle
Classifying testing by degree of
internal knowledge
●
Black box
●
White box
●
Grey box
Black box testing
White box testing
Grey box testing
Classifying testing by level
●
Unit testing
●
Integration testing
●
System testing
●
Acceptance testing
Unit testing
Integration testing
System testing
Acceptance testing
A/B testing
But wait…
How do carry out testing in a way that applies to
the CAMS principles?
Things we need to do
●
Make tests:
–
Automated
–
Repeatable
–
Implemented as code
–
Easy to change
Unit tests have traditionally been code
anyway...
Fuzzing
Useful principle when automating tests
Can we automate acceptance testing?
Can we automate acceptance testing?
Partially!
Test automation tools
●
Common test automation tools:
–
Selenium WebDriver
–
Capybara
–
Puppeteer
Selenium example
Puppeteer example
Puppeteer details
●
●
●
Written in TypeScript, runs on top of Node
Uses Chrome, requires an existing Chrome or
Chromium install
Runs Chrome in headless mode (i.e. without a
GUI)
Other types of tests
●
Regression testing
●
Non-functional testing
●
Smoke testing
●
Production canaries
Profiling
●
A form of dynamic program analysis – it works by
observing a program as it runs
●
Gathers detailed statistics about a program
●
Useful for performance monitoring, finding bugs, etc.
●
Usually requires a specialised build of the source
code to work (an instrumented build)
Flame graph
Flame Graph
mysql..
mysq..
mysqld`btr..
mysql..
mysqld`btr..
mysql..
mysqld`row_sel_get_..
mysqld`row_search_for_mysql
mysqld`ha_innobase::general_fetch
mysqld`ha_innobase::index_next_..
mysqld`handler::read_range_next
my..
mysqld`handler::read_multi_range..
mys..
mysqld`QUICK_RANGE_SELECT::get_n.. mys..
mysqld`find_all_keys
mysqld`filesort
mysqld`create_sort_index
mysqld`JOIN::exec
mysqld`mysql_select
l..
mysqld`handle_select
l..
mysqld`execute_sqlcom_select
l..
my.. mysqld`mysql_execute_command
libc.. my.. mysqld`mysql_parse
mysqld`dispatch_command
mysqld`do_command
mysqld`handle_one_connection
libc.so.1`_thrp_setup
libc.so.1`_lwp_start
m..
m..
mysq..
mysqld`eval..
mysqld`sub_select
mysqld`do_select
mys..
m..
mysql.. my..
mysql.. my..
m..
mysqld`row_.. mysqld`ro..
mysqld`row_search_for_mysql
mysqld`ha_innobase::general_fetch
mysqld`ha_innobase::index_prev
mysqld`join_read_prev
Search
mys..
mysq..
mysq..
mysq..
m..
m..
my..
mysq..
That’s all for this week
Thanks for your attention!
Introduction to DevOps
Lecture 11
Container orchestration
Eoin Rogers
eoin.rogers@tudublin.ie
Microservices
Microservices
●
Each microservice implements a small part of the full system
●
Each microservice runs in a container of it’s own
●
●
●
They communicate using well-known network protocols: this
allows for a very high degree of seperation of concerns
If one container goes down, it ought to be possible to restart it
without interfering with any other parts of the system
If you need to scale up the system this is easy: just run more
containers! Generally, we can have multiple containers of each
type running in parallel
Problems with microservices
●
●
But you need to create a lot of infrastructure to
make this work!
–
Set up a network between the containers
–
Start each container running
–
Monitor each container, and restart it if something
goes wrong
This is a lot of work!
Kubernetes
A container
orchestration platform:
it can automate most of
the things listed in the
previous slide for you!
First, we need some physical
computers
Worker nodes and control planes
Kubernetes basics
Kubectl
A client application that can be
used to give instructions to the
control plane from a remote
machine.
You use Kubectl to give
declarations outlining what you
want (not direct instructions)
and Kubernetes figures out the
rest!
Pods
Pods with multiple containers
Managing a cluster
●
Kubernetes runs a range of servers at both node and
control plane level to allow pods to be managed and
controlled
–
ReplicaSet a group of one or more pods of the same type,
running concurrently on the system
–
Service a resource that provides a network endpoint, so
pods can talk to each other, and reach the external
network
Data storage
●
●
●
Volume a resource that stores data – bascially a
folder that is shared between containers
ConfigMap a key-value store (generally implemented
as a JSON object) that can hold configuration data
Secret similar to a ConfigMap, but intended for
things that should be kept private (e.g. encryption
keys, login credentials etc.)
Deployments
●
An abstraction over ReplicaSets
●
This is generally what you work with in practice
●
Deployments create and manage a ReplicaSet
automatically. They have better support for rolling
updates, which are done by slowly scaling down a
deprecated ReplicaSet and scaling up a newer one
Labels and selectors
●
Pods aren’t directly linked to the resources that use
or manage them
–
●
For example, a ReplicaSet or Deployment won’t maintain
a list of its pods
Instead, pods store key-value metadata called labels
–
Resources find pods by running queries on the existing
labels in the system. These queries are called selectors
Example Deployment
We want to deploy 3 copies of a
container that is already on
Docker Hub with the name
somevendor/my-container
That’s all for this week!
Thanks for your attention!
Introduction to DevOps
Lecture 12
Infrastructure as Code and Cloud Computing
Eoin Rogers
eoin.rogers@tudublin.ie
Monorepos
Kubernetes
Vagrant
We can also run things on the
cloud?
This core idea is called infrastructure as code
(IaC).
This isn’t anything new!
We’ve been using IaC the whole time: Vagrant,
Docker, Kubernetes, Vagrant and Packer are all
concrete examples of Insfrastructure as Code!
There are other common IaC packages that are
worth knowing about...
AWS CloudFormation
●
Infrastructure as Code system built in to AWS
●
Integrates tightly with AWS
●
But only works with AWS, which might be a
problem!
Terraform
Terraform
●
Made by HashiCorp (same people who make Vagrant!)
●
Open source
●
Vendor neutral
●
Works with:
–
AWS
–
Google Cloud
–
Microsoft Azure
–
OpenStack
–
And more!
Terraform example
Puppet
Puppet
●
●
●
Both an open source and a paid, proprietary
version available
Developed by a private company called Puppet,
Inc.
Also widely used
Chef
Chef
●
Open source
●
Runs on a client-server model
●
Users include:
–
Facebook
–
AWS
●
Supports a wide range of platforms
●
Uses a dialect of Ruby to describe infrastructure
Core ideas behind IaC
●
●
●
IaC allows infrastructure (servers and the software
running on them) to be represented using a domainspecific formal language
This can be commited to a repository, so everyone
in the organisation has access to it
Machines can be created, changed or destroyed just
by running the IaC software
Cloud computing
●
●
I think everybody knows what cloud computing is :-P
Cloud computing is now the dominant technology for
deploying things in the real-world:
–
Some systems run 100% on cloud systems (e.g. Spotify)
–
Even big web companies use cloud to scale things up
and down quickly (e.g. Netflix, Facebook/Meta)
The old way of doing things
The new way of doing things
Advantages of cloud computing
●
Quicker to get started
●
Have the ability to scale up and down quickly
●
Often cheaper (except at really large scales)
●
“Cattle not pets” approach to server management
–
Works really well with IaC and the associated
philosophy!
Disadvantages of cloud computing
●
●
●
You’re entirely at the mercy of a big company
May be expensive at enormous scales (few
organisations are affect by this, however)
You don’t have access to the physcial hardware
(pulling the power cable is never an option!)
That’s the end of the course!
Thanks for your attention!