1.1 – Explain Linux boot process concepts
Boot Loaders


GRUB Legacy – menu used to select between kernels and OSes – Uses temp boot volume /boot
to store kernels and the GRUB config
GRUB2 – current version – only has one config file grub.cfg vs grub legacy having two, uses UUID
to identify disk, supports LVM and RAID, supports more OSes
Boot options





UEFI/EFI – causes boot problems for Linux, particularly with Secure Boot. The /boot/efi directory
is there to assist with UEFI systems for booting. If using EFI, a FAT32 partition is mounted on
/boot/efi containing boot files for various OS boot options with .efi extension
PXE – network-based booting – requires a PXE NIC and DHCP server with PXE
NFS – network File System – uses PXE to boot from a pre-built filesystem over TFTP, can be a
diskless workstation
ISO – can store an ISO image on a disk, many ways to configure. Can use as rescue disk on
separate partition, just need to edit grub.cfg file to point menu entry to ISO file
HTTP/FTP – similar to PXE, uses dedicated service like BFO/MKO(two separate projects) via ipxe
File locations






/etc/default/grub – GRUB2 settings
/etc/grub2.cfg – symbolic link to grub.cfg
/boot – contains all boot related info files and folders such as grub
/boot/grub – a GRUB directory
/boot/grub2 – a GRUB2 directory
/boot/efi – required if UEFI/EFI for various OS boots
Boot modules and files



Commands
o mkinitrd – creates initial RAM disk / image of filesystem as /dev/ram which is mounted
as temporary root filesystem for kernel to use(early user temp filesystem) during boot
process. The initrd process is very old, runs into problems if disk is changed(swapping
disk, changing size/partitions)
o dracut – involved in construction of intramfs image. Relies on udev(kernel’s device
manager) to detect devices and mounting
o grub2-install – installs grub onto a device
o grub2-mkconfig - scans for installed bootable OSes and generates a GRUB2 config file
initramfs – Uses initial root filesystem unpacked from an archive file(cpio type) and is mounted
in RAM as temp root filesystem. Generic and compatible with a lot of systems since no need to
have special modules precompiled or included in kernel
efi files – boot loader executables, exist on UEFI based computers, contain data on how boot
process proceeds


vmlinuz – the Linux kernel executable – compressed kernel that can load the OS into memory to
the comp becomes usable and apps can run
vmlinux – uncompressed version of kernel image that can be used for debugging
Kernel panic


Major issue occurs and kernel is designed to ‘panic’ and usually either completely stops or
reboots itself, also attempts to write to logs
Some causes for kernel panics:
o Corrupted hard disks
o Incomplete kernel modules
o File/directory permissions issues
o Invalid memory references or memory addresses containing key codes
o Hardware failure including ram or CPU failure or unhandled bugs
o System overheating
1.2 Given a scenario, install, configure, and monitor kernel modules
Commands







lsmod – lists all active kernel modules(system-level software used directly by OS kernel) – has no
options
insmod – inserts a module into the kernel at a low level – doesn’t read from set location or
manage dependencies – can use module-specific options
modprobe – inserts modules at a higher level into kernel – used in conjunction with depmod to
determine and manage dependencies
modinfo – used to extract module information for installed modules
dmesg – access kernel ring buffer(fixed size buffer) containing messages about kernel modules
being loaded – useful to see recent device loads even during boot process
rmmod – removes a module from memory – doesn’t remove dependency modules(i.e. if all
modules depending on module A are removed, module A still exists)
depmod – updates a database to maintain current info about modules
Locations




/usr/lib/modules/[kernellversion] – directory for modules per kernel version
/usr/lib/modules – see above, related
/etc/modprobe.conf – older file used in previous distros used t modify how modules are loaded
and unloaded
/etc/modprobe.d – newer version of the above in directory format housing files that modify
how modules are loaded and unloaded
1.3 Given a scenario, configure and verify network connection parameters
Diagnostic tools

ping – verify a remote host can respond to a network connection. -c is common flag












netstat – used to display network information. -r to display routing table, -s for statistics, -ta to
display all active TCP connections
nslookup – queries DNS servers, deprecated
dig – queries DNS servers
host – used to perform hostname-to-IP translation, i.e. DNS queries
route – used to view the routing table, can use to modify/add routers
ip – replaces some commands, ex: “ip addr show” is similar to ifconfig’s output
ethtool – used to display and configure network device settings, i.e. transmission speed/duplex.
Useful options: -I to get device driver info, -S statistics, -p identify device by making link light
blink, speed/duplex/autoneg
ss- used to display socket information
iwconfig – display info about wireless networks. Useful parms: essid/nwid/mode/ap
nmcli – configures NetworkManager, tool designed to detect/configure network connections
brctl – used to work with Ethernet bridges
nmtui – provides a text-based interface to configure NetworkManager
Configuration files









/etc/sysconfig/network-scripts/ - Red Hat based distros - directory containing files used to
configure network devices. Most common one is ifcfg-interface
/etc/sysconfig/network – Red Hat based distros – file containing basic networking info
/etc/hosts – file containing hostname-to-IP-address translations, used usually for local network
/etc/network – location for Debian-based network files. Primary file is /etc/network/interfaces
/etc/nsswitch.conf – used to get name resolution, either host file or DNS server usually
/etc/resolv.conf – list of DNS servers for the system
/etc/netplan – new location for Debian distros to store network config files. YAML format.
/etc/sysctl.conf – specifies which kernel settings to enable at boot
/etc/dhcp/dhclient.conf – used to configure a DHCP server
Bonding




Purpose is to have system treat multiple network interfaces as if they were a single interface,
providing greater bandwidth. Seven modes available.
Aggregation – putting two or more network interfaces into a single cluster. Can be done
manually or automatically(requires config file)
Active/Passive – refers to backup modes either active or passive. Active-backup is one device
being active and others are backups if active device goes down
Load Balancing – allows system to share traffic among network devices based on some criteria
1.4 Given a scenario, manage storage in a Linux environment
Basic partitions





Partitions are used to separate a hard disk into smaller components, can be treated as different
storage devices. Previously limited to four partitions but now can do more than four and
separate filesystems can be created
o Primary partition – contains installed OS and is bootable
o Extended partition – not bootable and usually contains data and logical partitions(i.e.
/dev/sda4 may be an extended and used as container for logical partitions /dev/sda5 &
/dev/sda6
Older partition tables were MBR while newer ones are usually GPT which doesn’t have same
limitations as MBR
Raw devices – a device file associated with a block device file(hard disk, etc). Provides direct
access to the block device. Does not have a filesystem
GPT – designed to overcome MBR limitations. Can have 128 partitions per hard disk device
MBR – traditional partitions, limited to four partitions by default
File system hierarchy




Real Filesystems – placed on physical storage device, i.e. partition, software RAID device, LVM
logical volume. Ex: ext3, ext4, xfs
Virtual Filesystems – consists of a collection of real filesystems merged together via folders
called mount points. Root directory / is the starting point of the VFS.
Relative paths – file or directory path in relation to the current directory
Absolute paths – file or directory path in relation to the root directory /
Device mapper



LVM – addresses issues with regular partitions – reg partitions are not resizable, LVM provides
ability to chance size of structures called logical volumes – reg partitions can’t exceed disk size
but LVM can take several phys devices and merge together to create a larger logical volume –
LVM also provides ‘snapshot’ ability to make backups easy
o Example: You have three 1TB hard drives and want to use more than 1TB of data as one
“drive”. LMV will first have you create three 1TB physical volumes, then those volumes
get added to a volume group, then from that volume group you can, for example, create
a 2TB hard drive from the total 3TB available
o /dev/mapper directory contains ‘virtual’ devices that are symbolically linked from the LV
device, i.e. /dev/lg1/lv1 > /dev/mapper/lg1_lv1
mdadm – Command used to create RAID devices
Multipath – allows for configuration of multiple network paths to storage devices
Tools

XFS tools
o xfs_metadump – dumps unmounted XFS filesystem metadata into a file used for
debugging



o xfs_info – used to display geometry of an XFS filesystem
LVM tools
o vgremove – deletes a VG. VG must not have any LVs.
o vgreduce – deletes a PV from a VG
o vgextend – adds a PV to an existing VG
o vgdisplay – displays info about a VG
o pvdisplay -displays info about a PV
o lvdisplay – displays info about an LV
o lvextend – extends the size of an LV
EXT Tools – see below for mkfs, dumpe2fs, resize2fs, tune2fs
Commands
o mdadm – used to create a software RAID device
o fdisk – used to display and modify non-GUID(i.e. MBR) partition tables. The -l option
displays the table, i.e.: fdisk -l /dev/sda
o parted – used to display/modify GUID and non-GUID partition tables. Can also create a
filesystem on a partition. Use -l to display info like with fdisk.
o mkfs – create a filesystem on a partition
o iostat – provides I/O statistics on devices, including partitions. The -d command is useful
to display info.
o df – displays usage of partitions and logical devices. -h is a useful option.
o du – provides an estimated amount of disk space usage in a directory structure. -h is a
useful option
o mount – displays current mounted filesystems as well as mounts filesystems manually
o umount – unmounts a filesystem
o lsblk – displays device locations and relevant info(including mount points)
o blkid – display labels and UUIDs
o dumpe2fs – display filesystem metadata for a device/partition
o resize2fs – usually used to resize a logical volume
o fsck – finds filesystem problems on unmounted filesystems
o tune2fs – displays or modifies specific metadata for an ext2/ext3/ext4 filesystem
o e2label – changes the label of a filesystem
Location










etc/fstab – used to display which filesystems to mount, where to mount the filesystems, and
what options to use during mounting
etc/crypttab – describes encrypted block devices that are set up during boot
/dev – displays device files
/dev/mapper – displays logical volumes
/dev/dis/by-id – displays devices by serial number of hardware
/dev/dis/by-uuid – displays devices by UUID
/dev/dis/by-path – displays devices based on hardware path to device
/dev/dis/by-multipath - devices displayed based on the multipath
/etc/mtab – list of mounted filesystems, done automatically
/sys/block – describes block systems, i.e. hard drives, DVDs, LVM devices


/proc/partitions – kernel’s partition table
/proc/mounts – list of all mounted filesystems
Filesystem types







ext3 – extension of the ext2 filesystem, designed to be placed on disk-based devices(partitions).
Major difference between 2 and 3 is addition of journaling.
ext4 – replacement for ext3 filesystem. Supports larger filesystem and individual file sizes,
better performance
xfs – disk-based filesystem known for high performance and large file sizes
nfs – network based filesystem, old, originated with Unix. Newer versions have better
performance.
smb – Samba filesystem, based on cifs, provides network sharing
cifs – used in Microsoft Windows to share folders across the network
ntfs – used in Microsoft Windows disk-based systems. Linux cannot recognize ntfs.
1.5 Compare and contrast cloud and virtualization concepts and technologies
Templates






VM – a definition of a VM that contains information about the VM
OVA – VM template saved in OVF format but stored in TAR format. Not supported by all
hypervisors. End in .ova extension. Used to distribute VM solution
OVF – Open Virtualization Format – standard defining a template package including: descriptor
file in XML format describing VM components/settings; Disk image; Additional resources
JSON – JavaScript Object Notation – Used by virtualization/cloud-based technologies to store or
transfer data; uses object, array, string and number data formats
YAML – similar to JSON, uses indentation instead of characters like {} and []
Container Images – less overhead than VMs, typically ran as applications on host OS, still uses
images. Examples: Docker, Container Linux, cgroups
Bootstrapping




Refers to any process that self-starts without the need for external assistance
Cloud-init – Provides initialization of a cloud instance. Makes use of pre-provisioned images.
Typical operations – add SSH keys, perform networking configs, execute scripts
Anaconda – installer program, can automate Kickstart
Kickstart – Contains all answers to Anaconda questions during installation, no human
intervention required. Can also do post-installation tasks like installing additional software
Storage

Thin vs Thick Provisioning – Thick: solution in which all requested storage space is provisioned
when devices is created, can create wasted space. Thin: volume space is requested but only a
portion is initially allocated, more effective use of storage space



Persistent volumes – storage space that can be utilized by an OS. Persistent means OS is not
aware of storage device changes as new devices get the volume space reallocated to it if
changed
Blob – binary large object – data stored as a single object within a DB
Block – physical storage device typically back-ended for cloud-based storage systems, ex: SSD,
traditional SATA drives
Network Considerations





Bridging – combining multiple network segments into one segment. Types: Simple(connects two
segments, Multiport(connects multiple networks), Transparent(learning bridge, builds routing
tables dynamically), Source route(Routing built by one source of traffic)
Overlay networks – built on top of another network, can happen when two different network
protocols are used. Network packets encapsulated within other network packets. Ex: Tor, VPN
NAT – Network Address Translation – private network packets translated so they can be routed
to public network
Local – LAN – collection of systems that can communicate directly with each other
Dual-Homed – a system that has more than one network interface for redundancy
Types of Hypervisors


Native hypervisors – Type 1 – Bare metal, directly interact with system hardware, no OS
middleman – Hyper-V KVM, RHE Virtualization, VMWare Xen
Hoste hypervisors – Type 2 – OS runs them – VirtualBox, VMware Player
Tools



libvirt – provides an interface between hypervisor and kernel
virsh – component of libvert software, allows you to manage virtual machines, i.e. autostart –
have guest domain start automatically when OS starts; reboot – reboot a guest domain; save –
save the current state of guest domain
vmm – virtual machine manager – i.e. virt-manager is GUI tool to manage KVM VMs
1.6 Given a scenario, configure localization points
File Locations


/etc/timezone – location of system time zone in Debian based systems
/usr/share/zoneinfo – contains a list of all zone files
Commands




localectl - displays and changes both locale values and keyboard layouts
timedatectl – display system clock
date – display system clock – options available for customizable format
hwclock – display the real-time clock
Environment variables




LC_* - collection of locale settings used to change the way the shell and other programs handle
differences based on geographic region of the user
LC_ALL – can be set to override all other locale settings
LANG – can be set to provide a default locale value, can be overwritten for specific locale
features by setting other locale variables, i.e. setting default to Spanish but use date/time in
English
TZ – can be used to set different time zone than system default
Character Sets



UTF-8 – see below with Unicode
ASCII – English-only encoding format limited to 128 characters. Extended ASCII supports nonEnglish characters
Unicode – standard that includes ASCII within first 7 bits(128 characters). Additional bits used
for non-English characters
2.1 Given a scenario, conduct software installations, configurations, updates
and removals
Package Types





.rpm – Red Hat package file, managed with rpm command
.deb – a Debian package file, managed with dpkg command
.tar – file created with the tar command, an archive
.tgz – compressed tar archive performed with gzip command
.gz – compressed file created with gzip command
Installation tools






RPM – useful dealing with .rpm package installation, upgrades, and removal
dpkg – used to manage Debian packages
APT – use apt-get to manage Debian packages located on a repository, has many options and
commands to work with it; apt-cache displays package information regarding package cache;
aptitude – menu driven tool to add/remove packages
YUM – install software from repositories – Red Head based, lots of options similar to APT;
yumdownloader downloads RPM files without installing, can be done manually later;
/etc/yum.conf is primary yum config file; /etc/yum.repos.d contains .repo files with repo
locations
DNF – designed to replace yum, commands work just like yum mostly except duf; handles
dependencies better
Zypper – found in SUSE Linux, derived from RPM, works similar to yum, auto dependence check
Build tools



Commands
o make – uses a file named Makefile to perform operations; utility for building and
maintaining programs and other types of files from source code; determines which
pieces of a large program need to be recompiled and issues necessary commands to do
so
o make install – install code from source on the system, may include compile process
o idd – displays the shared libraries used by specific commands, i.e. ldd /bin/cp; used to
troubleshoot code being written
Compilers – utility that takes source code and converts to executable binary code, ex: gcc
compiler
Shared Libraries – files used by executable programs, designed so devs can rely on established
code to perform functions; naming convention libname.so.ver.
Repositories




Configuration – yum/apt tools used to do this
Creation – createrepo package used with yum; dpkg-dev used with APT
Syncing – used to duplicate existing repo on local system either to act as standalone or mirror
repo; YUM repo uses reposync, APT uses apt-mirror
Locations – accessed using FTP or HTTP
Acquisition commands


wget – noninteractive tool to download files from remote systems via HTTP, HTTPS, or FTP
curl – noninteractive data transfer from large number of protocols, supports more protocols
than wget, can’t do recursive downloads or recover from failed download attempts like wget
can
2.2 Given a scenario, manage users and groups
Creation


useradd – used by root to create a user account; several options available
groupadd – used by root user to create a group account
Modification




usermod – used by root user to modify a user account
groupmod – used by root user to modify a group account
passwd – allows a user to change their password; root can use to change any user’s password
chage – used by root user to modify password aging features for a user account
Deletion

userdel – used by root user to delete a user account

groupdel – used by root user to delete a group account
Queries





id – displays basic account information; without arguments it displays current users UID,
username, primary GID name, and secondary group memberships
whoami – displays the effective user ID
who – shows who is currently logged in; includes username, terminal device in use, login
date/time, where user is logged in from
w – displays who is logged in as well as other useful info
last – displays info about current and previous logins
Quotas



Allows admin to limit how much disk space can be used by individuals or groups
User quotas – filesystem must be mounted first with usrquota mount option, remount
afterwards
o After mounting, a quota DB must be created using quotacheck command
o edquota allows creation or editing of user’s quotas
o quota command displays quotas for a user account
o repquota displays quotas for entire filesystem
o quotaon turns on quotas for a filesystem, quotaoff turns them off
Group quotas – affects groups and is similar in use to user quotas
Profiles


When a user logs into a system, a login shell is started. If a new shell is started after login that is
a non-login shell. Initialization files are used to setup the shell environment depending on the
shell type(login or non-login) Fig. 8.1/8.2 for breakdown
Bash parameters – used to modify bash shell features
o User entries – users can control their individual environment modding these files
 .bashrc – when user opens a new BASH shell commands in this file execute;
allows users to setup account by placing commands in this file
 .bash_profile – when user logs in and the login shell is BASH, commands in this
file are executed
 .profile – same as above
o Global entries – admin can control BASH environment for all users by using these files
 /etc/bashrc – when a user opens a new BASH shell the commands in this file are
executed; only root can modify this file
 /etc/profile.d/ - devs may need to modify BAHS environment of user accounts,
i.e. dev uses shell variables to pass programs key information such as account
info or system info; devs can modify this directory by adding files here
 /etc/skel – used when a new user account is created to provide the new account
with default files
 /etc/profile – when user logs in and login shell is BASH shell the commands here
are executed; only root user should modify this file
Important files and file contents



/etc/passwd – stores user account info including except password and password-aging policies
/etc/group – stores group account info
/etc/shadow – stores user password info; passwords are encrypted
2.3 Given a scenario, create, modify, and redirect files
Text Editors


nano – non-GUI editor providing a handy cheat sheet at the bottom of the screen; commands
given holding down Ctrl + another key
vi – standard with virtually all distributions; a bit of a learning curve, uses command and insert
modes as well as ex mode(for more complex operations); replaced effectively by vim
File readers






grep – searches files for lines containing a specific pattern, powerful command with lots of
options
cat – displays the contents of text files
tail – displays the bottom portion of text data; 10 lines are the default
head – displays top portion of text; 10 lines are the default
less – displays larger chunks of data, similar to cat but more interactive
more – predecessor to less, less capable
Output redirection












commands are able to send two streams of output(standard output and standard error) and can
accept one stream of data(standard input); STDOUT, STDERR, STDIN; STDOUT/STDERR go to
terminal window by default and STDIN come from keyboard by default, but they can be
redirected with different operators
< - override STDIN so input comes from a specified file
> - override STDOUT so input goes to a specified file
2> - override STDERR so output goes to a specified file
&> - override both STDOUT and STDERR so output goes to specified file
| - also called ‘piping’, output of one command goes to input of another command; ex: ls
command results can go to grep to filter output
<< - see Here Document
>> - if a file doesn’t exist it is created, if it exists the information on STDOUT gets appended to
the end of the file
/dev/null – redirecting here results in information on STDOUT/STDERR not showing up
anywhere; a black hole effectively; bit bucket
/dev/tty – describes virtual terminals; accessed sitting directly at system, can use Ctrl-Alt-F1 to
access /dev/tty1, and F2, F3 etc
xargs – takes info from STNDIN to craft execute commands; ex: ‘ls -d /etc/e* | xargs wc -l’ runs
wc -l on every file in the /etc directory beginning with the letter e
tee – can be used to send STDOUT to both the terminal and a file

Here Documents – redirection of a large chunk of data to a command; ex: ‘cat << EOF “sample
text” EOF’ will send ‘sample text’ to the cat command as if it came from STDIN
Text Processing









grep – see above
tr – translates characters from one set to another; ex: date | tr ‘a-z’ ‘A-Z’ – capitalizes date
output
echo – displays information STDOUT; ex: ‘echo $PATH’ displays path variable value
sort – used to sort text data; each line separated into fields with whitespace as default delimiter
awk – modifies text in a simple DB format
sed – command line ‘text editor’ effectively; automates modifications to files
cut – used to display sections of data
egrep – performs same function as ‘grep -E’ command
wc – displays number of lines, words, or characters of data
File and directory operations



















touch – creates an empty file and updates the modification and access timestamps of existing
file
mv – move or rename a file
cp – copy files or directories
rm – delete files or directories
scp – copy files to and from remote systems via Secure Shell
ls – list files in a directory
rsync – useful to copy files remotely across the network
mkdir – creates a directory
rmdir – delete empty directory; used with -r will recursively delete a non-empty directory
ln – link files
o Symobolic(soft) – original file contains data and link file points to original file; changes
made in original file will also appear to be in linked file; link file points to filename only,
not the actual inode table, link file has its own inode table; when a process follows the
link from link file, the data is accessible via the link file
o Hard – two filenames pointing to the same inode and hence the same data; inode table
points to blocks within storage device where data is stored
unlink – does similar task to rm, deletes a file; lacks options like rm has
inodes – information such as owner/permissions stored in filesystem element called inode;
everything about a file except the data itself is stored in the inode
find – searches live filesystem for files or directories using different criteria; many, many options
locate – searches for files based on a DB usually created daily
grep – see previous
which – locates various executables
whereis – searches for binary executables, source code, and manual pages
diff – compares two files
updated – used to update the DB used by the locate command
2.4 Given a scenario, manage services
Systemd management

Systemd is a newer method of system initialization, replaces SysVinit system
o Instead of init scripts running serially, they run in parallel with one another
o Not as dependency driven as SysVinit
o Core binaries in system suite system – systems and services manager for operating system
 systemctl – system state inspection and controlling utility
 system-analyze – used to inspect performance stats regarding system bootup as
well as view trace and current state info
o Targets are used instead of runlevels like in SysVinit; Google for runlevel/target
equivelancies; each target has specific services that start
systemctl









used to administer system-based distros
Enabled – setting used to start a service at boot time
Disabled – setting used to make it so that a service won’t start a boot time
Start – start a service not currently running
Stop – stop a service currently running
Mask – makes it impossible to start/enable a service; used when a conflicting service can’t or
shouldn’t be removed from system
Restart – restarts a service not currently running
Status – display status of service
Damon-reload – used in conjunction with sytemd.generator to restart it; system.generator
creates components and dependency trees and automatically creating unit files
Systemd analyze blame

systemd-analyze – used to determine if there are any errors in Systemd config as well as to
debug Systemd; many options available but the blame option is important for exam; output of
command displays all currently running services and how long it took for each service to
successfully load
Unit Files


unit files are used to define a service; ‘systemctl list-units’ displays unit files defined on system
o .service file has three sections usually: [Unit], [Service], [Install]
o [Unit] – Describes the service, includes man file location
o [Service] – specifies what process should be started as well as some options/config
o [Install] – includes WantedBy setting which indicates which target wants to start this
service, i.e. if printer.target is engaged then this service gets started if it lists
printer.target in the WantedBy field
Directory locations
o /lib/systemd/system – standard location for Systemd files
o




/etc/systemd/system - alternative location for Systemd files; most files created by
admin go here
o /run/systemd/system – contains runtime units; essentially units created on the fly
Environment parameters – some units require environment parms to function correctly; can be
set in a unit file’s EnvironmentFile setting
Targets – see above
Hostnamectl – used to view and change host/system information
Automount – process of mounting a resource when a process access the mount point, when
process stops using mount point the resource is unmounted; i.e. CD-ROM, USB disk; create
automount by creating unit files i.e. a .automount and a .mount; execute systemmctl daemonreload afterwards
SysVinit





Init process is started by the kernel, reads /etc/inittab to determine the default runlevel;
/etc/rc.d/rc.sysinit boots to single-user mode then /etc/rc.d/rc boots the system the rest of the
way to the specified runlevel; scripts in /etc/rcX.d folders starting with letter ‘K’ are processed
with the ‘stop’ parameter and scripts starting with ‘S’ are with the ‘start’ parameter depending
on runlevel
Chkconfig – used to display or modify when services are started or stopped; --list option displays
runlevels and services and what status they are per runlevel
o on – turn a service on; ex: chkconfig Bluetooth on
o off – turn a service off; ex: chkconfig Bluetooth off
o level – lets you specify levels to turn on and off services
Runlevels – init is the first process the kernel starts and it starts all additional services depending
on the runlevel the system is booted to; runlevel is a functional state in a system
o 0 – halt the system
o 1 – single-user mode
o 2 – multi-user mode without networking or GUI
o 3 – multi-user mode with networking no GUI
o 4 – not used
o 5 – Multiuser with GUI
o 6 – Reboot
o /etc/init.d – actual scripts from /etc/rcX.d folders since they’re symlinked
o /etc/rc.d – contains runlevel folds and initial rc.sysinit script
o /etc/rc.local – can be used to place commands that will execute during normal bootup
o /etc/inittab – config file that determines the default runlevel to boot to
Commands
o runlevel – to see current runlevel
o telinit – change system to different runlevel
Service
o Restart – stops and starts a service
o Status – shows the current status of a service
o Stop – stops a service
o Start – start a service
o
Reload – reloads a service’s config file; useful when changes are made but don’t want to
restart service due to potential adverse effects
2.5 Summarize and explain server roles
NTP




Designed to resolve issues regarding system time of server and client systems
Important for log files, client-server interactions, searching for a file by timestamp, transaction
log timestamps
System setup as an NTP client and receives time based data from one or more NTP servers;
usually 3+ NTP servers used
Stratum-0 clock is based on atomic clockwith minimal delay, stratum-1 gets its time from
stratum-0
SSH


Service that allows for secure communications between hosts; includes symmetrical and
asymmetrical encryption and hashing
Used as secure replacement for telnet, secure replacement for ftp and rcp; sftp and scp
Web

Provides website based services as well as DB access and mail service
Certificate Authority

A server that verifies digital certificate signatures
Name Server

Provides service to translate hostenames to IP addresses
DHCP

Provides client systems with network information like an IP address, gateway IP address, name
server and network mask
File Servers

A system that shares files
o FTP server – Provides very little security, not always a terrible thing; i.e. sharing files
over the internet with an anonymous user account configured to prevent uploads;
requires FTP client to access
o SFTP – SSH used to provide FTP-like access but in a secured manner; secures data
transport; requires SSH client program
o Samba – shares both files and printers; made with Microsoft Windows in mind; good
solution in an environment with multiple OSes
o NFS – primarily for sharing files between Linux/Unix systems; usually used only on a
LAN; accessed via a mounted NFS share in the filesystem(setup by administrator)
Authentication Server

Used to authenticate user accounts on client systems; ex: Radius – used for AAA services, IPA –
combo of different technologies(LDAP, Kerberos, NTP, DNS, CA, etc)
Proxy


A go-between amid two systems used for:
o Monitoring network traffic and filtering based on content of the traffic
o Bypassing filters by appearing to be originating from a different system/network
o Can log network traffic and remote system access
o Improving performance by caching static data
o Can provide security features
o Can translate data from one language to another
Most common server on Linux is Squid
Logging



The process of taking reports form services and storing the data into files or databases for future
retrieval
Syslog is a common logging service ran as syslogd; usually stored in /var/log directory; new
version is rsyslogd
Also journald used in conjunction with rsyslogd and is a component of systemd; displayed via
journalctl
Containers

A lightweight VM essentially, can be used for running solo applications, don’t require their own
components for the most part like virtual CPU/RAM/HDD; docker is popular
VPN

A service that allows for a virtual network over a public network transferring data securely
Monitoring

Wide variety of monitoring software available, i.e. Wireshark/tcpdump, Nagios, psacct/who/w
Database

System used to store and return data upon request, a lot of options available; accessed via SQL;
managed via Oracle/MySQL/PostgreSQL etc
Print Server

Manages print jobs; most common for Linux is CUPS
Mail Server

Responsible for sending/receiving/storing email messages; utilizes SMTP to interact with other
mail servers; Sendmail is popular, Postfix is more common in Linux as it is easier to configure
and more secure
Load Balancer

Utility that spreads work across multiple servers or systems; heavy loads require more than one
server and this does the job of triaging requests
Clustering

Grouping together a collection of computers to provide a specific service; can be used to
provide lots of processing power, spread load across different systems, provide huge storage
space
2.6 Given a scenario, automate and schedule jobs










cron – allows you to schedule processes to run at specific times; uses crond daemon which
checks every minute to see what process should be executed
at – lets you schedule a time to execute a certain command, i.e. “at 5pm tomorrow” gives you
an at> prompt to enter a specific command to run at 5pm tomorrow; atq command lists current
user’s at jobs; /etc/at.deny and /ect/at.allow is used to filter what commands a user can run
crontab – allows user to view or modify her crontab file; crontab file allows user to schedule a
command to be executed on a regular basis; important options: -e to edit file, -l to list file, -r to
remove all entries from file
o /etc/cron.deny – all users listed in this file are denied access to crontab command
o /etc/cron.allow – all users listed in this file are allowed access to crontab command
o Neither file exists – only root can use crontab command
o Both exist – only /etc/cron.allow is consulted
fg – A paused process can be restarted in the foreground with this command
bg – paused process can be restarted in the background with this command
& - added to end of command to execute and run in the background
kill – change the state of a process; commonly used to terminate a process, by itself allowing
process to cleanly exit; using -9 forces process to close without being clean
ctrl-c – use this to kill a process running in the foreground
ctrl-z – used to pause a process, then fg/bg can be used to restart
nohup – prevents the HUP(hang-up) signal from stopping a child process; used if you need to
disconnect from a system but want a process to continue since disconnected causes HUP to be
sent to all processes you started by default
2.7 Explain the use and operation of Linux devices
Types of devices

Bluetooth – designed to communicate wirelessly over a very short range(30m or so); useful for
headphones/keyboard/mice












WiFi – designed to communicate wirelessly but has larger bandwidth and range than Bluetooth
USB – interface provides for wired communication; mice/keyboards etc
Monitors – used to display output of laptop or desktop system; config files in /etc/X11
GPIO – General purpose I/O; programmable pin on a chip; example found on Raspberry Pi –
contains a row of GPIO to connect different programmable devices
Network adapters – allow network communication via wired connection
PCI – slot on motherboard; network and audio cards; lspci displays attached PCI devices
HBA – card or circuit board that connects to motherboard usually via PCI slot
SATA – used for mass storage devices; CD-ROM/DVD/HDD/SDD are SATA now usually
SCSI – used for mass storage devices; rare on laptops/desktops anymore; may be on high-end
servers
Printers – Allows you to make a physical copy of a digital file or photo
Video – graphics card; allows computer to communicate with monitor
Audio – allows computer to communicate with speakers/headphones
Monitoring and configuration tools










lsdev – shows devices recognized by the system
lsusb – displays usb devices that are attached to PCI bus
lspci – displays devices attached to the PCI bus
lsblk – displays block devices attached to the system
dmesg – displays the in-memory copy of the kernel ring buffer(a data structure that records
messages related to the operation of the kernel)
lpr – see CUPS section
lpq – see CUPS section
abrt – see CUPS section
CUPS – can be configured via command-line, we-based or GUI-based tools; exam focuses on
command line
o lpadmin – used to create a printer
o cupsctl – used to perform CUPS configuration, displaying current settings by default
o /etc/cups directory is the location of the CUPS configuration files
o lpr – sent a print job to the printer
o lpq – displays status of a printer; -a displays status of all printers
o abrt – would remove print jobs from print queue in older versions
udevadm – userspace management tool for udev
o add – see udev section for adding udev rules
o reload-rules – ‘udevadm control –reload’ will reload new rules for udev
o control – see above
o trigger – verify a new rule after a device has been created
File locations


/proc – provides info regarding processes, kernel features, and system hardware; memory
based/not stored on hard drive
/sys – provides important info regarding devices and buses that the kernel is aware of



/dev – contains device files; used to access physical and virtual devices; memory based/not
stored on hard drive
/dev/mapper – used with LVM to assist with assigning device names
/etc/X11 – xorg.conf is primary config file for X server
Hot pluggable devices




udev is the Linux kernel’s device manager; manages devices under /dev using info from sysfs; if
device is plugged in while it is running udev can configure it and make it available to the system;
these devices are considered hot plug devices
/usr/lib/udev/rules.d – System Rules – Lowes Priority – should not be modified
/run/udev/rules.d – Volatile Rules – rules created/destroyed dynamically by
applications/daemons
/etc/udev/rules.d – Local Administration – Highest Priority – udev device manager is managed
by config files in this directory; can include entries that modify the behavior of a device file such
as ownership or permissions placed on device file
2.8 Compare and contrast Linux graphical user interfaces
Servers



In Linux a GUI server is the software that provides the interface between the hardware and the
software
Wayland – More lightweight and simpler structure than X11 with newer features
X11 – legacy system created in 1984 by MIT; current source version maintained by X.Org
Foundation; includes drivers to work with hardware components like mouse/monitor; provides
GUI operations like draw/resize/move GUI apps on screen; does not provide an actual desktop
GUI






Desktop environment defined by having – File manager, control panel/center, window manager,
common look/feel, integrated office suite, media players
Gnome – only includes FOSS software, simple and easy to use; ex distros: Fedora/Debian
Unity – uses existing desktop apps with focus on interface; intended for Ubuntu
Cinnamon – newer and designed initially for Linux Mint to replace Gnome; focus on
graphics/effects/animations
MATE – newer desktop, originally extension of Gnome, more lightweight than others
KDE – highly customizable, has many tools
Remote desktop




VNC – very popular in Linux and Microsoft Windows, origin early 2000s, VNC server gets started
then VNC client connects
XRDP – based on FreeRDP, can be used on Microsoft Windows to connect to Linux desktop
NX – cross-platform, flexible and has nice features, can be used on native and virtual installs
Spice – acquired by Red Hat in 2009, Linux specific, some virtualized environment support
Console redirection


The process of SSH port forwarding
SSH port forwarding – also called SSH tunneling – provides the means to create secure
connection between two systems using software provided by SSH
o Local – SSH forwarding originates from client machine – outgoing port on local system
configured to connect via SSH to specific port on remote system
o Remote – imagine a remote web server and also one hosted locally, remote SSH allows
for an end user to connect to IP of remote web server which then opens SSH tunnel
back to local web server, hiding local IP address
o X11 – allows SSH client programs the ability to receive data from GUI based programs to
display on local X server
o VNC – remote desktop, see above
Accessibility

Examples:
o Sticky Keys – allows user to do things like Ctrl-c without holding down modifier button
when pressing the combo
o Repeat keys – continuously inserts a key value when a user holds down a key on
keyboard
o Slow keys – must hold a key down for a short period of time to get it to show on screen
o Mouse keys – can move mouse cursor with keys on keyboard
o Braille display – monitor that provides braille output
o Onscreen keyboard – self explanatory
3.1 Given a scenario, apply or acquire the appropriate user and/or group
permissions and ownership
File and directory permissions





Read, write, execute – files/directories associated with read/write/execute permissions for
owner, group, and other users; 10 characters in the ls -l output denotes the permissions and file
designation: first space if blank is plain file, if d is directory, next 9 in groups of three are
owner/group/other users; w = write, r = read, x = execute
User, Group, other – see above
SUID –
o suid – when set on executable files, suid allows a pgoram go access files using
permissions of the user owner of the file; chmod u+s or chmod 4xxx
o sgid – when set on executable files sgid allows a program to access files using
permissions of the group owner of the file. When set on directories all new files in
directory inherit the group ownership of the directory; chmod g+s or chmod 2xxx
o sticky bit – if set on directories it makes it so files in directory can only be removed by
user owner of file, the owner of the directory, or root user; chmod o+t or chmod 1xxx
Octal Notation – see chmod below
umask – sets default permissions for files/directories; applied when initially created




Sticky bit – see above
GUID – see above
Inheritance – Linux permissions don’t utilize inheritance, i.e. new files/directories don’t inherit
permissions from the directory that item is created in
Utilities
o chmod – used to change permissions on files; octal method – permissions assigned
numeric values: read = 4, write = 2, execute = 1; permissions set like ‘chmod 754
filename’ means file is rwxr-xr—; symbolic method – u/g/o/a and +/-/= and r/w/x
combinations to set/remove permissions
o chown – used to change user owner or group owner of a file or directory; ex: ‘chown tim
abc.txt’ will change ownership of abc.txt file to tim user; useful options -R for recursive
change, -v for verbose
o chgrp – changes group ownership of file, same usage as chown above
o getfacl – see below
o setfacl – used to set an access control list(ACL) for a file or directory; format is: ‘setfacl option what:who:permission file/directory’; getfacl reads the ACL report generated by
setfacl
o ls – can be used with -l to display long output with permissions listed
o ulimit – lists or sets a user’s account limits; common limits: fsize = max file size allowed
in memory, cpu = max CPU time allowed, nproc = max number of concurrently running
processes, maxlogins = max number of concurrent logins
o chage – used to determine amount of time between password changes
Context-based permissions



SELinux configurations
o Sets up SELinux “contexts” which are similar to security groups; most admins use
SELinux to secure processes that may be compromised by hackers making use of
exploits; ex: hacker compromises Apache server and uses a /usr/sbin/httpd process
spawned by apache ‘user’ to access /etc/passwd file contents – SELinux can lock down
processes so they can only access a certain set of files
o disabled – when in this mode SELinux is not functional at all, no checks performed when
users attempt access files
o permissive – SELinux checks but doesn’t block access to files/directories; used for
logging and troubleshooting
o enforcing – SELinux performs checks and blocks access to files/directories as necessary
SELinux policy
o targeted – targeted policies contain rules designed to protect the system from services
rather than regular users
SELinux tools
o setenforce – used to set different policy modes
o getenforce – to determine current SELinux mode
o sestatus – provides overall status information about SELinux
o setsebool – used to set an SELinux Boolean
o getsebool – used to see Boolean settings; Boolean is either true/false value
o
o

chcon – change the context of a file or directory
restorecon – used to restore the default security context of a file or directory; -R will do
it recursively
o ls -Z – used to see a security context for a specific file
o ps -Z – used to see security context for running process
AppArmor –
o A MAC(mandatory access control) system that is similar to SELinux
o aa-disable – used to disable an AppArmor profile(rule set describing how to restrict a
process)
o aa-complain – mode that has AppArmor report problems
o aa-unconfined – lists processes not restricted by AppArmor profiles
o /etc/apparmor.d/ - location of definitions of AppArmor profiles
o /etc/apparmor.d/tunables – location of files used to fine-tune AppAarmor behavior
Privilege escalation





su – allows user to shift user accounts; used with – option spawns new login shell
sudo – allows users(when appropriately configured) to run commands as other users(typically
root uers); must be configured in /etc/sudoers file
wheel – a group commonly used to allow non-root root access(if enabled in /etc/sudoers file)
visudo – used to edit /etc/sudoers and it does formatting checks
sudoedit – can edit a file using sudo access with this command; chosen editor depends on
variables(SUDO_EDITOR, VISUAL, EDITOR)
User types



Root – root account is the system admin account; UID of 0; has full system control
Standard – account with UID of 1000 or higher
Service – usually UID under 1000; some are referred to as daemon accounts using daemonbased software
3.2 Given a scenario, configure and implement appropriate access and
authentication methods
PAM

Pluggable Authentication Modules – used by almost all Linux utilities to attempt to authenticate
users. Examples of things PAM can do:
o Can be used to enforce more robust password requirements
o Limit days/times users can log in to the system
o Limit locations users can log in from
o Can set or unset environment variables; can have one set of variables for local login, one
for SSH logins, one for FTP logins, etc
o Can restrict user accounts
o Limit where the root user can log in from







Password policies – the pam_unix module provides lots of features that modify how passwords
are set, ex: save last five passwords for a user and disallow reuse of those
LDAP integration – add lines to /etc/pam.d/system-auth file to add LDAP; Google for reference
User Lockouts – see below in pam_tally2 and faillock
Required, optional, or sufficient – in a PAM config file the values are columned by: type(account,
auth, etc), control value, and then the PAM module to use. Control values:
o required –
 Returns successful – if the current PAM module returns successful, the next rule
is checked in the list unless it’s the final rule, then the stack returns successful
 Returns unsuccessful – no additional modules are executed, stack returns
unsuccessful
o requisite –
 Returns successful – same as for required
 Returns unsuccessful – additional modules are executed; however, regardless of
what happens with these other modules, the stack returns with value of
unsuccessful; useful as other modules may provide more info in log files
o optional –
 Returns successful – next rule is checked in the list; if final then stack is
successful
 Returns unsuccessful – next rule is checked in the list; if final stack is
successful(unless it’s the only rule in the stack, then the stack is unsuccessful)
o sufficient –
 Returns successful – no additional modules are checked
 Returns unsuccessful – next rule is checked in the list, if final the stack is
successful
o include – tells PAM to use all the rules from the file specified
/etc/pam.d – each file here is designed to configure a command or utility that uses PAM to
authenticate user accounts
pam_tally2 – can be used to lock a user out after unsuccessful login attempts
faillock – same as above, different config file
SSH


Allows you to connect to a Secure Shell service; encrypts data in a tunnel as opposed to
unsecured telnet
~/.ssh/ - SSH data for individua users is stored here; used by SSH to store important data; users
can modify configurations in this directory
o known_hosts – client stores a server’s unique fingerprint key in this file after a
connection is established with an SSH server; typically left alone
o authorized_keys – location where public keys are stored in key-based ssh authentication
after manually copying keys over from client to server
o config – place where you can customize how commands like ssh, scp, and sftp work
o id_rsa – used for password authentication in conjunction with the ssh-agent and ssh-add
utilities
o






id_rsa.pub – used for password authentication in conjunction with ssh-agent/ssh-add
utilities
User-specific access – refers to using passwordless authentication using public/private keys
TCP wrappers – uses simple config files to either allow or deny access from specific
hosts/networks; only services that use the TCP Wrappers library will be affected the
/etc/hosts.allow and /etc/hosts.deny files
/etc/ssh – contains files that can be used tto administer both the SSH server and SSH client
utilities
o ssh_config – used to modify the behavior of the SSH client utilities like ssh/scp/sftp;
affects all users but users can override these settings by creating ~/.ssh/config
o sshd_config – used to configure the SSH server; many options in this file that can be
configured
ssh-copy-id – used to copy login keys to a remote system
ssh-keygen – used to generate authentication keys
ssh-add – used to add RSA/DSA encryption keys to the SSH agent’s cache; ex: use it after sshagent to cache keys
TTYs



A device file associated with a terminal display which is traditionally a command-line login
screen
/etc/securetty – lists all device files in which the root user can log in to the system; identified by
the /etc/tty# device names in the file
/dev/tty# - TTY device files named /dev/tty# where # is actually a number; typically only seven in
use on standard Linux distros; one reserved for GUI-based logins
PTYs

Pseudo-terminals; these are provided to a shell when a user logs in remotely(SSH) or when a
new terminal window in a GUI-based environment is invoked

Public Key Infrastructure – used to ensure a server is really where user intended to go as
opposed to a rogue server; often provides means to encrypt data between server/user
TLS – asymmetric cryptography – uses public and private keys to encrypt/decrypt data
Public key encrypts data sent to Apache Web Server; freely given to the web browser; server
decrypts with private key
Digital signature(message digest) is sent to a Certificate Authority to verify the signature, and
this goes on the certificate
Digital certificates typically have an expiration date
See diagram below
PKI





VPN as a client






Similar to SSH in that it provides secure data transfer; uses public/private keys to
encrypt/decrypt data; ex: OpenVPN
SSL/TLS – protocol used by VPNs to provide secure transport of data, TLS is more common and
SSL is deprecated; commonly used in web server comms, email transport and VoIP
Transport mode – only data is encrypted, IP information not encrypted; common with site-tosite VPN
Tunnel mode – both data and metadata are encrypted; common with client-to-site VPN
IPSec – used in VPNs for authentication and to encrypt network packets; performs actions at OSI
Layer 3, whereas TLS/SSL operate above Layer 3
DTLS – utilizes datagram-based security features; datagram is used on packet-switched
networks as basic unit of transfer; uses connectionless communication method
3.3 Summarize security best practices in a Linux environment
Boot Security


Boot loader password – Boot loader can allow user to perform custom operations during boot
process including booting to alternate kernels or runlevels; best practice to enable bootloader
password
UEFI/BIOS password – UEFI/BIOS can allow custom booting operations – best practice to enable
password
Additional authentication methods


Standard method is local user/pass combo; lacks more complex security features
Multifactor Authentication – Uses multiple bits of evidence for proof of identity like ‘Something
you know”, Something you are”, “Something you have”
o Tokens – unique value generated by hardware device or software program
 Hardware – typically small device on key fob, generates token
 Software – generated by program, like an app on mobile device
o OTP – one-time password; can be used with 2FA or if user loses password




o Biometrics – fingerprints, retina scanner, etc
RADIUS – protocol that allows client system to authenticate via server, provides
AAA(Authentication, Authorization, Accounting); managed centrally
TACACS+ - similar to RADIUS; Cisco created
LDAP – protocol providing directory services information, can store info like hostnames;
RADIUS/TACACS+ is more robust
Kerberos – uses a ticket-based system; server grants a ticket-granting-ticket(TGT) upon
authentication, encrypts and the secret key is sent back to client system who then can use it to
authenticate to other services
o kinit – used to obtain individual ticket
o klist – display list of cached Kerberos tickets
Importance of disabling root login via SSH


Usually for servers directly accessible to the Internet as hackers can try to directly login as root;
disabling this capability requires the hacker to compromise a non-root account and then
through that they would have to compromise the root account
Modify /etc/ssh/sshd_config file by saying ‘PermitRootLogin no’
Passwordless login


Normally associated with SSH and is convient/security feature; uses public/private key pair
Enforce use of PKI – best practice when using key pairs for SSH logins
Chroot jail services

A service that only has access to a small portion of the filesystem such as its subdirectory so that
it won’t be able to see or modify critical system data; ex: BIND
No shared IDs

If multiple users use same ID(UID) their actions can’t be accounted for; don’t do this
Importance of denying hosts

Stems from the fact that there are systems known to be used as launching points for hacker
attacks; these hosts should always be denied access to your systems; some sites provide a list
for these hosts; ex: https://mxtoolbox.com
Separation of OS data from Application data


Avoid allowing an application fill up filesystem with log files(as an example); partitioning is the
best way to prevent this
Disk partition to maximize system availability – see above
Change default ports

Changing default port for a network service can deter/prevent hackers from attacking well
known ports(i.e. SSH uses port 22, change this to another port)
Importance of disabling or uninstalling unused and unsecure services





FTP – commonly used protocol to transfer files between systems – not encrypted, including
user/pass so if someone can snoop the network they can see user/pass; anonymous FTP doesn’t
require authentication, only use for download, not upload
Telnet – permits remote login without encryption – don’t use this
Finger – provides info about computers/users; long-time use to provide a report on a user;
unencrypted; don’t use in modern systems
Sendmail – see below
Postfix – Sendmail/Postfix are email servers; no need on a local system; configure a real email
server instead; disable or limit this and other services(CUPS is an example of printing isn’t
needed)
Importance of enabling SSL/TLS

Used by HTTPS – provides secure manner of connecting for web services
Importance of enabling auditd

Used to log user account activity; determine if user has performed any unauthorized activity
CVE monitoring

Provides info about publicly known vulnerabilities; admin should monitor relevant reports and
implement fixes for vulnerabilities asap
Discouraging use of USB devices

Can be used to inject or steal data from servers; common to disable USB devices in BIOS
Disk encryption


Makes it nearly impossible to access data unless unencrypted; only encrypted while system off
LUKS – disk encryption method common in Linux systems; uses kernel module dm-crypt
Restrict cron access

Code can be executed on critical systems; restrict access
Disable ctrl-alt-del

Used to prevent rebooting servers when they shouldn’t be
Add banner

Some services like FTP/SSH/web servers can let you add a banner for users to see when they
first sign in; can be informational or warning about only authorized access
MOTD

Displayed when user first logs in; provides useful info about system being used
3.4 Given a scenario, implement logging services
Key file locations




/var/log/secure – file contains log entries related to authentication and authorization
operations; includes when users log in, attempts to gain escalated privileges, etc
/var/log/messages – wide variety of log entries from numerous services/daemons; i.e. crond,
kernel, mail servers
/var/log/[application] – most applications today create their own logs without needing a logging
service
/var/log/kern.log – contains messages from the kernel
Log management




Third-party agents – syslogd/rsyslogd – logs application and system events; /etc/syslog.conf is
config file
logrotate – ensure the partition that holds the log files has enough room to handle them;
rotates log files to limit filesystem space that the log uses; /etc/logrotate.conf is config file
/etc/rsyslog.conf – one of the config files for syslogd
journald – stores log entries in a special file format in order to optimize log file entries
o journalctl – handles logging process on modern Linux systems; can be used to query
systemd log entries
lastb

displays failed login attempts
3.5 Given a scenario, implement and configure Linux firewalls
Access control lists









Aka firewall; designed to allow or block network traffic
Source – where network packets originate from; can be filtered from source using IP/MAC
address/Port
Destination – where a network packet is being sent to; filters can be applied here
Ports – unique number used to address a service on a system; packets contain
source/destination port; can be filtered based on this
Protocol – i.e. ICMP/TCP/UDP or telnet; can be filtered this way
Logging – can be enacted by rules; useful for later packet inspection
Stateful vs Stateless – Stateful applies to any previously established connection; stateless applies
regardless of prior established connection
Accept – Once a network packet matches the criteria of a firewall rule, a target is used to
determine what taction to take(Accept/Reject/Drop/Log) – Accept tells iptables to allow packet
and advance to next filtering point
Reject – tells iptables to return the packet to source with an error message; not advanced


Drop – tells iptables to discard packet; no response sent to source; not advanced
Log – tells iptables to create a log entry about packet; neither allowed nor blocked, other rules
in rule set may determine result
Technologies




firewalld- Red Hat based distros – used to configure iptables rules – managed by firewall-cmd
command
o Zones – rules are configured into categories called zones – internal/trusted zone on
network may have less restrictive rules while Internet(dmz/external zone) may be more
restrictive
o Run Time – when rules are created using firewall-cmd command they affect the active
firewall on the system, this is run time firewall. If system is rebooted or firewall service
restarted rules are lost
iptables* – rule sets(chains) are applied at different places(filter points), allowing flexibility;
types of rules(table) can be placed at a filtering point; filter points can have more than one set
of rules; iptables can do multiple functions: filter data, perform NAT operation on packet,
mangle packet; once a rule is matched a target is performed(accept/drop/reject/log)
o Persistency – changes made by iptables command only affect currently running firewall;
can be made persistent using iptables-save command
o Chains – see above
ufw – Debian based distros – front-end interface to create iptables rules
Netfilter – component of kernel that performs NAT and IP forwarding
IP forwarding

kernel feature allowing network packets to be passed from one network to another; used to
create a router on a server; set the value of the below files to 1:
o /proc/sys/net/ipv4/ip_forward – enables IP forwarding for IPV4 network packets
o /proc/sys/net/ipv6/conf/all/forwarding – enables IP forwarding for IPV6 network
packets
Dynamic rule sets




Some utilities dynamically create firewall rules to thwart hacking attempts on the fly
DenyHost – designed especially to protect SSH servers; creates TCP Wrapper rules; ex: if remote
system attempts brute force attack DenyHost creates a blocking rule in /etc/hosts.deny
Fail2ban – scans specific log files searching for IP addresses that attempt breaches by repeated
connection attempts; config file /etc/fail2ban/jails.conf
IPset – designed to create sets of IP addresses and then use this set to apply rules to collection
of systems
Common application firewall configurations


/etc/services – location where services are mapped to ports(traditionally)
Privileged Ports – ports 1-1023 reserved for commonly used protocols
3.6 Given a scenario, backup, restore, and compress files
Archive and restore utilities



tar – purpose of tar command is to merge multiple files into a single file
cpio – used to create archives
dd – used for backing up data and creating files; commonly used to backup an entire drive
Compression




gzip – used to compress files
xz – used to compress files
bzip2 – used to compress files
zip – used to merge multiple files into a single, compressed file
Backup types





Incremental – includes all files that have changed since last incremental or full backup
Full – everything from source is backed up; very time consuming but restoring from this is
quicker
Snapshot clones – frozen image of the filesystem; used with LVM; used to safely back up a live
filesystem
Differential – archives any files since last full backup
Image – refers to the format used for the backup of data; ex: tar, ISO
Off-site/off-system storage



SFTP – uses SSH to securely transfer files across the network
SCP – copy files to and from remote systems via Secure Shell
rsync - useful to copy files remotely across the network
Integrity checks



Useful to determine if a file has been changed or replaced with an imposter
MD5 – hashing algorithm creating a unique message digest that can be used to verify a file; if
the file changes, the digest will also change
SHA – similar to MD5 but uses a different algorithm
4.1 Given a scenario, analyze system properties and remediate accordingly
Network monitoring and configuration









Latency – delay communication either one-way or round-trip
o Bandwidth – maximum amount of data that can travel through media(network cable,
wireless, etc); netstat is a helpful command here
o Throughput – actual amount of data passing through media(this is not the same as
bandwidth, but bandwidth is a factor of this); netstat/Wireshark helpful here
Routing – the process of sending network data from one IP network to another via a gateway;
gateway is a system that has multiple network interfaces and transfers the data between the
networks; the route and netstat commands are helpful here
Saturation – occurs when throughput often reaches the value of the bandwidth; can be a
problem and cause sluggish networks; Wireshark is good here
Packet Drop – when a remote system doesn’t respond to an inbound network packet; firewall
rules/saturation/misconfigured network could be the culprit
Timeouts – software component used to determine if a packet drop happens; the ping
command is a simple example here, if packets aren’t getting replies
Name resolution – when a hostname is translated into an IP address; dig/host/nslookup
commands provide this
Localhost vs Unix socket –
o Localhost is the system communicating with a special localized network interface;
network traffic sent to localhost(127.0.0.1) doesn’t go on the network, and is kept
internally; Wireshark can help analyze traffic on local machine
o Unix socket is a method allowing processes to communicate on the same system with
one another; difficult to troubleshoot since network-sniffing tools cant be used here
Adapters – referring to network adapters
o RDMA drivers – allow network adapters to communicate between each other without
the need for the OS to get involved; downside is no confirmation of communication
receipt
Commands
o nmap – used to probe a remote system to determine which network ports are reachable
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
netstat – displays network statistics to determine if there is an issue with transmitting or
receiving data
iftop – provides a display of the network connections using the most bandwidth
route – displays routing table to determine if a route to another network is correctly
configured
iperf – used to create tests of throughput between two systems; client-server model
tcpdump – displays all network traffic to standard output until command is terminated;
packet sniffer used to view local network traffic
ipset – creates a set of IP addresses that can be used by other tools like iptables
Wireshark – network sniffer that comes with GUI or TUI interface(tshark); very popular
netcat – also referred to as nc; a very large toolset that can do virtually anything related
to TCP or UDP; man page has good breakdown of functions
traceroute – “traces” the pathway a network packet follows to a remote system; useful
to determine where a bottleneck or roadblock occurs during communication
mtr – a variation of traceroute that performs the action once per second and displays
statistics constantly being updated
arp – used with the ARP table, displays MAC address/IP address information
nslookup – display results of hostname-to-IP-address lookup
dig – similar to nslookup
host – similar to dig/nslookup
whois – used to determine which company/person owns a domain
ping- determine if a remote system can be reached via the network
nmcli – displays network settings in order to identify misconfigured interfaces
ip – displays network device settings; useful to determine misconfigured interfaces
tracepath – similar to traceroute but doesn’t require superuser privileges and has no
fancy options
Storage monitoring and configuration








iostat - provides I/O statistics on devices, including partitions. The -d command is useful to
display info.
ioping – used to perform latency tests on a disk
IO scheduling – the kernel uses a scheduler to optimize IO; several parameters can be used to
affect this behavior
o cfq – has a separate queue for each process, and each queue is served in a continuous
loop
o noop – follows first in/first out principal
o deadline – standard scheduler; creates a read queue and a write queue
du - provides an estimated amount of disk space usage in a directory structure. -h is a useful
option
df - displays usage of partitions and logical devices. -h is a useful option.
LVM tools – see previous section regarding LVM tools commands
fsck – finds filesystem problems on unmounted filesystems
partprobe – used in situations where partition table has changed and system needs to know
about these changes
CPU monitoring and configuration





/proc/cpuinfo – contains defailed information about CPUs
uptime – shows how long the system has been running
loadaverage – displays system load average; see uptime
sar – shows same info as iostat; displayed over time in 10 minute intervals
sysctl – see IO Scheduling
Memory monitoring and configuration








swapon – displays currently active swap devices; can be used to add swap devices to currently
used swap devices; /etc/fstab can be used to use a swap device each time boot occurs
swapoff – used to remove a device from current swap space
mkswap - sets up a Linux swap area on a device or in a file
vmstat - reports information about processes, memory, paging, block IO, traps, and cpu activity
Out of memory killer – used by the kernel to kill a process using the most memory in the event
the system runs out
free – provides a summary of virtual memory
/proc/meminfo – provides info to the free and vmstat commands
Buffer cache output – vmstat displays this information; information regarding file contents
stored temporarily in memory
Lost root password


Steps to remedy this issue
o Reboot system to single user mode
o Mount root filesystem
o Manually edit the /etc/shadow file and remove the root password
o Reboot system and log in as root user(won’t require password)
o Set the root password
Single user mode
o Level in which only the root user can log in; limited functionality and no GUI; used to fix
boot problems or recover root password
4.2 Given a scenario, analyze system processes in order to optimize
performance
Process management

Process states – processes are assigned states based on what current actions they are taking;
use the ps and top commands to display states
o Zombie – a process that has been terminated but hasn’t been entirely cleared out of
memory
o Uninterruptible sleep – process is performing certain system calls that prevent it from
being interrupted(killed); rare
o Interruptible sleep – process that is performing some kind of I/O operation, i.e.
accessing the hard disk; common




o Running – currently taking place on CPU or has operations in CPU queue
Priorities – indicated by ‘nice’ values; tells CPU which process has higher priority; range from -20
to +19 where -20 is highest priority, 0 is default, +19 is lowest priority
Kill signals – via kill command; used to change state of a process including stopping(killing) it;
determine process ID first, then use kill command i.e. ‘kill 17844’; normal kill command gives
process time to cleanly stop, if -9 option is used this forces process to stop without time to clean
up; there are a handful of common signals that the kill command can send to impact
processes(SIGHUP, SIGINT, SIGKILL, SIGTERM, SIGTSTP)
Commands
o nice – specify a different nice value other than default; regular users can’t assign a
negative value, only root
o renice – change nice value on existing job
o top – displays process info updated on a regular basis regarding the processes using the
CPU the most(by default); interactive mode allows actions to be taken on displayed
values
o time – used to determine how long it takes to run a command
o ps – used to list processes that are running on the system
o lsof – lists open files; can be useful if used with -i option to show files related to open
network connections
o pgrep – a combination of ps and grep commands
o pkill – can be used to kill processes by things like names, usernames, or group names
PIDs – unique number assigned to each process on a system
4.3 Given a scenario, analyze and troubleshoot user issues
Permissions


File – look closely at related error and then check output of ‘ls -l’ to see permissions, noting the
owner and the group; verify the user’s username with the id command and match against file
permissions
Directory – permissions work differently than file but same general rules as file permission
troubleshooting apply
Access


Local – physically sitting at the workstation and logging in directly; problems include hardware
malfunctions, entering incorrect user account information, PAM restrictions
Remote – connecting to the system via the network
o TCP Wrapper rules issues – see prior section about TCP Wrappers
o Misconfiguration of the service – i.e. SSH may be misconfigured or not running
o Service-based security restrictions – i.e. SSH may not allow root user to log in directly
o Network-based issue – if remote system not accessible via network, can’t log on
o Firewall restrictions – remote access may be blocked
Authentication



Local – typically relies on password stored in /etc/shadow file; problems include user forgetting
password, account passed expiration date, password aging restrictions, admin manually locked a
user account, PAM restriction limiting user authentication; things to do include check security
log for error messages, review user account settings in /etc/shadow file, review PAM restrictions
External – this is handled via LDAP, see prior section about LDAP or other reference guide for
more detailed info
Policy violations – typically defined by PAM, check PAM config files like /etc/pam.d/sshd to
ensure directives are properly set, check pam_tally2 or faillock utility to see if account is locked
File Creation




Quotas – users may be limited in ability to create new files if quotas enabled on a filesystem;
system admin can use repquota command to display quotas for users; regular users can use
quota command to see limitations on their own account
Storage – if filesystem is full, new files can’t be created in filesystem; df command useful here
Inode exhaustion – files have unique inodes to keep track of metadata, filesystem has limited
number of inodes, if this is exhausted it could cause inability to create files(this is rare
nowadays); --inodes option with df command is helpful
Immutable files – files that cannot be changed by anyone, including root user; chattr command
can be used to do this; example is /etc/passwd file
Insufficient privileges for authorization
SELinux violations – see prior chapter about SELinux to determine if violations have occurred
Environment and shell issues
Big consideration here is user’s shell variables and how they are configured; pay attention to PATH
variable as this is a common reason why commands may not work without explicit/complete pathname
4.4 Given a scenario, analyze and troubleshoot application and hardware issues
SELinux context violations


Can be triggered by incorrect policy configurations; check audit log file with sealert
command(install this via setroubleshoot package)
Mislabeled files can cause problems here as well; check context with ls -Z; if context needs to be
changed can do chcon utility to modify it, semanage to make it permanent, restorerecon to fix
labels
Storage


Degraded storage – devices fail over time; searching for bad blocks and using the SMART tool via
the smartctl command can shed some light on possibly degraded storage devices
Missing devices – can be several causes: removable device may not be attached or may not be
powered on, internal device may have loose connection, could simply be broken, kernel
adapters(modules) may not be loaded into memory





Missing mount point – mount point must exist prior to mounting a partition or volume, create it
with mkdir command
Performance issues – can check performance with dd command: ‘dd if=/dev/zero of=/var/test
bs=1G count=1 oflag=direct’; can also be related to cache which can be activated with ‘hdparm
-W1 /dev/sda’ and deactivated with ‘hdparm -W0 /dev/sda’
Resource Exhaustion – can mean several things:
o Device has no more available space to create more files; use df command to check
o Device is out of inodes(necessary for metadata storage); df command can check this
o User is limited by disk quotas
Adapters – a kernel module used to access a device; info can be seen via lspci -v command
o SCSI – device type used to attach storage devices to a system; lsscsi command is helpful
here
o RAID – device drivers can be viewed with lspci -v, more info can be viewed with lsraid
command
o SATA – device type used for storage, check drivers with lspci -v
o HBA – used to attach various devices to a system, commonly for Fibre Channel or SAS
 /sys/class/scsi_host/host#/scan – can be modified to enable a hot-plug SCSI
device
Storage integrity
o Bad blocks – use badblocks command to test for bad blocks; also SMART tool can locate
them
Firewall



Restrictive ACLs – check for ACLs with ‘drop’ or ‘reject’ targets; pay attention to ‘denied access’
responses in the case of ‘reject’ targets; no response is provided with drop target ACLs
Blocked ports – First determine the port number being used by a service to send the packet,
keeping in mind standard vs non-standard ports; look for rules blocking these ports
Blocked protocols – First determine protocol being used(ICMP, TCP, UDP), then check ACL rules
Permissions





Ownership – Consider the following:
o User/owner of a file or directory has ability to change permissions
o User/owner also has a set of permissions separate from all other users
o Group owners have a set of permissions that other users don’t share
o User and group ownerships can be modified; chown or chgrp commands
Executables – A file that has execute permission set; critical for files needing to be run like a
program
Inheritance – Linux files/directories don’t utilize inheritance
Service accounts – typically locked, don’t have usual login shell, usually a service account per
service for security purposes
Group memberships – see above regarding group accounts and permissions
Dependencies






Normally yum/apt handle dependencies(dnf also), but sometimes these have to be handled
manually
Patching – the process of updating software; different organizations have different patching
policies
Update issues –
o Access to the repository may not be available
o Permissions may prevent update installation; root access normally required
o Update may contain bugs, breaking the software or entire OS
o Update could overwrite previous config files; back up all config data before upgrading
o Update may overwrite existing data files or DBs; back your stuff up
o Update may cause program to behave differently; ensure communications about
changes are made to users
Versioning – software version included in the package name; don’t install an older one than is
currently in the system
Libraries – shared libraries can be linked to any program at run-time once loaded in memory
Repositories – keep these up to date and know which ones contain the required software
Troubleshooting additional hardware issues









Memory – consider compatibility issues with memory sticks; use the ‘free’ command to monitor
memory usage as well as vmstat; prior section outlines further memory related issues like ‘Out
of Memory Killer’
Printers – most issues come down to printer drivers and CUPS; others may be related to paper
jams, working with the printer queue and invalid print jobs, prior section outlines some of this
Video – normally stems from broken hardware devices or compatibility issues; check with
website of X server for possible new drivers
GPU Drivers – GPUs relate to graphics cards and typical issues are centered around drivers; keep
them current
Communications ports – in this context relates to I/O ports for devices like keyboards/mice;
display I/O ports via /proc/ioports; also can check lsdev for hardware info
USB – use lsusb command to see info about USB devices attached to your system; check
/var/log/messages or /var/log/syslog to see info about recently attached USB devices
Keyboard mapping – use xev to perform keyboard matching operations
Hardware or software compatibility issues – move device to another system to see if issue
persists; try another similar hardware device, reinstall the software related to the hardware;
upgrade the hardware drivers
Commands
o dmidecode – used to display a description of hardware components
o lshw – produces quite a lot of info about system hardware
5.1 Given a scenario, deploy and execute basic BASH scripts
Shell environments and shell variables

Shell variables store information that is used to modify shell behavior or external commands









When a variable is created it’s only available in the shell where it was created, and is called a
local variable; variables can be passed into subprocesses via the export command
PATH – variable containing paths containing executable programs; scripts should be put in these
directories
Global – variables defined outside of the shell script at the main shell level; inherited by the
script shell environment
Local – defined within shell script; accessible only from within shell script
export – converts local variable to environment(global) variable; if variable doesn’t exist export
can create it as environment; can also display all environment variables with -p option
env – displays environment variables in current shell; can also be used to temporarily set a
variable for command execution
set – displays all shell variables/values when executed with no arguments; can also be used to
modify shell behavior; unset command removes a variable from the shell
printenv – used to display environment variables; fewer options than env command
echo – used to display information, typically the value of variables; i.e. ‘echo $HISTSIZE’
#!/bin/bash

For BASH shell scripts this should be the first line in the script as it is the path to the interpreter
Sourcing scripts

The source command executes bash scripting as if the commands within the script were
executed directly on the command line, since most commands are executed as a separate
process in their own environment
Directory and file permissions


Root user should not be the owner of scripts, only regular users as this can create security
issues; once a script is created use ‘chmod a+x script_name’ to add execute permission
chmod – used to update file/directory permissions
Extensions

Typical BASH extension is .sh, can use .bash though; .pl = Perl, .py = Python, .tesh = TC, .csh = C
Commenting

Use # to create comments in scripts to help others read and better understand what different
portions of the script do
File globbing

File globs(wildcards) represent a portion of a filename; i.e. * which matches zero or more
characters in a filename
Shell expansions



${} – parameter expansion – can be used to fetch the value of a parameter or even temporarily
substitute its value
$() – command substitution – executing a subcommand within a larger command; used to
gather data and store in a variable; i.e. ‘today=$(date)’ stores the output of the date command
in the $today variable
` ` - used like $()
Redirection and Piping – see output redirection section further up
Exit codes


When a command executes it returns a success/failure value stored in the $? Variable; can be
used in conditional statements
stderr/stdin/stdout – discussed previously above
Metacharacters

Special characters in BASH shell, i.e. globbing characters(*,?, etc) and redirection like | > <
Positional parameters

Used to grab command line arguments when shell script is executed; $1, $2, $3 etc are used in
the script itself to denote which argument(s) to grab
Looping constructs




Used in scripts to perform actions potentially multiple times
while – perform tasks as long as conditional statement returns a value of true
for – used to iterate over a list of values
until – performs tasks until return value is false
Conditional statements



Used to compare values and perform functions depending on true/false of the condition
if – used to execute one or more commands based on outcome of a conditional statement
case – compares variable values to globbing patterns; format:
case $var in
glob 1)
#statements
#statements;
glob 2)
#statements
#statements;
Esac
Escaping characters

Having a special character treated as a plain character; i.e. ‘echo “this | that”’ treats the pipe
like a normal character and displays ‘this | that’;
o Double quotes disable special meaning of redirection/globbing characters
o Single quotes disable all metacharacters
o The backslash \ can be placed before a metacharacter to escape it
5.2 Given a scenario, carry out version control using Git
Arguments









clone – git clone creates a local repository from the contents of a remote repository
push – causes changes from local repository sent to remote repository
pull – causes changes from remote repository to be downloaded to local repository
commit – causes changes made to the working directory placed in the local repository; first add
to the staging area then commit to repository
merge – allows you to merge changes/updates made to software in a ‘test’ branch with the
current software in the master branch; must be in master branch to perform this function
branch – used to create a new branch; you will originally be in the master branch; use checkout
command to switch between branches
log – used to see changes made on different branches along with comments
init – used to create a new repository in current local directory
config – used to configure the git utility; --global options causes configuration options being
stored in config file in the user’s home directory
Files


.gitignore – used to have git commands ignore a file; create this file in working directory and
place the filename to ignore inside of the file
.git/ - local repository data stored in this directory under the directory the git command created
5.3 Summarize orchestration processes and concepts
Agent and Agentless




Orchestration is used to simplify complex setup tasks in which software or services require
multiple steps; allows for a single command to move through these steps
Monitoring is a component of orchestration in which a determination is made regarding
whether an action needs to take place due to something happening on the system
Agent monitoring – a piece of software has been installed on the system that is being
monitored; actively probes system determining if changes have been made requiring action
Agentless monitoring – no software installed on system; receives data from system or remote
queries are conducted to determine if a change has taken place
Procedures

The collection of steps that need to take place to complete an action; can be straightforward or
complex
Attributes

Used to define parameters that are used to customize the automation process; contain data
that will be different for each orchestration process
Infrastructure automation

The process of deploying an app and its environment through automated means
Infrastructure as code

Uses software tools to configure and provision systems
Inventory

Information regarding an software instance that is being orchestrated; often stored in a
database
Automated configuration management

Done to avoid unnecessary errors and remove time-consuming tasks when configuring
software/services/systems; promotes consistency and predictability; examples of tools are Chef,
Puppet, Ansible
Build automation

Done to alleviate errors and complexity when turning source code into executable code;
common build automation tools include Make, Ant, Gradle, Rake, and Cake