Table of Contents
Exercise 1 ............................................................................................................................. 2
Exercise 2. Text Viewer, The: less is More ............................................................................ 4
Exercise 3. Bash: The shell, .profile, .bashrc, .bash_history .................................................. 6
Exercise 4. Bash: working with files, pwd, ls, cp, mv, rm, touch ............................................ 9
Exercise 5. Bash: environment variables, env, set, export .................................................. 12
Exercise 6. Bash: language settings, LANG, locale, dpkg-reconfigure locales .................... 14
Exercise 7. Bash: redirection, stdin, stdout, stderr, <, >, >>, |, tee, pv ................................. 17
Exercise 8. Bash: more on redirection and filtering: head, tail, awk, grep, sed .................... 20
Exercise 9. Bash: job control, jobs, fg ................................................................................. 25
Exercise 10. Bash: program exit code (return status) .......................................................... 28
Exercise 11. Bash: wrapping up .......................................................................................... 30
Exercise 12. Documentation: man, info ............................................................................... 33
Exercise 13. Documentation: Google .................................................................................. 36
Exercise 14. Package management: Debian package management utility aptitude ............ 38
Exercise 15. System boot: runlevels, /etc/init.d, rcconf, update-rc.d .................................... 43
Exercise 16. Processes: working with proccesses, ps, kill ................................................... 48
Exercise 17. Job schedulers: cron, at .................................................................................. 53
Exercise 18. Logging, /var/log, rsyslog, logger .................................................................... 57
Exercise 19. Filesystems: mounting, mount, /etc/fstab ........................................................ 64
Exercise 20. Filesystems: modifying and creating filesystems, tune2fs, mkfs ...................... 68
Exercise 21. Filesystems: changing root directory, chroot ................................................... 75
Exercise 22. Filesystems: moving data around: tar, dd ....................................................... 76
Exercise 23. Filesystems: security permissions, chown, chmod, umask.............................. 81
Exercise 24. Networking: interface configuration, ifconfig, netstat, iproute2, ss, route ......... 87
Exercise 25. Networking: configuration files, /etc/network/interfaces ................................... 98
Exercise 26. Networking: packet filter configuration, iptables ............................................ 101
Exercise 27. Networking: secure shell, ssh, sshd, scp ...................................................... 114
Exercise 28. Performance: getting performance stats, uptime, free, top ............................ 123
Exercise 29. Kernel: kernel messages, dmesg .................................................................. 130
Exercise 30. Lather, Rinse, Repeat: The Grand Rote Learning ......................................... 133
1
Exercise 1
Now, log into a Linux virtual machine and type:
vim hello.txt
You should see the following:
Hello, brave adventurer!
~
~
~
~
~
~
~
~
~
~
~
~
~
"hello.txt" [New File] 0,0-1
All
There is a joke that says that vim has 2 modes – “beep repeatedly” and “break
everything”. Well, if you do not know how to use vim, it is pretty much true, because
vim is modal text editor.
The modes are:
NORMAL mode: moving your cursor around and doing operations with text like
delete, copy and paste.
INSERT mode: entering text.
This greatly confuses newcomers because they try to avoid NORMAL mode as
much as possible. Well, this is wrong, so now I will give you correct outline of
working with vim:
start vim
while editing is not finished, repeat
navigate to desired position in NORMAL mode
enter INSERT mode by pressing i
type text
exit INSERT mode by pressing <ESCAPE>
when editing is finished, type :wq
What is the most important is that you stay in NORMAL mode almost all the time,
entering INSERT mode for a short time to type something and then immediately
exiting it. Looking this way, vim has only one mode, and that mode is NORMAL
mode.
Now, let us try it out. Remember, press i to enter INSERT mode and <ESCAPE> to
return to NORMAL mode. Type the following (press <ENTER> at the end of each
line):
iRoses are red
Linux is scary
<ESCAPE>
This is what you should see:
Roses are red
Linux is scary
~
~
2
~
~
~
4,17
All
Now I'll give you list of commands to navigate the cursor in NORMAL mode:
h — move left
j — move down
k — move up
l — move right
i — enter INSERT mode
o — insert a line under cursor and enter INSERT mode
<ESCAPE> — exit INSERT mode
x — delete symbol under cursor
dd — delete a line
:wq – write changes to the file and exit. Yes, that's right, it is colon follower by wq
and <ENTER>.
:q! – do not write changes to the file and exit.
That will suffice. Now, place cursor on first line and type this:
oViolets are blue<ESCAPE>
After this, place cursor on the line “Linux is scary” and type this:
oBut I'm scary too<ESCAPE>
You should see this:
Roses are red
Violets are blue
Linux is scary
But I'm scary too
~
~
~~
~
~
~
4,17
All
Now type :wq to save the file and exit. You should see this:
Violets are blue
Linux is scary
But I'm scary too
~
~
~~
~
~
"hello.txt" 4L, 64C written
student@vm1:~$
Well, you did it! You just edited a text file in vim, the mighty and terrible!
Extra credit
Start vim again by typing vim hello.txt and try some other commands I gave you.
Play this game which will allow you to feel more comfortable in vim: http://vimadventures.com/
3
Exercise 2. Text Viewer, The: less is More
Now you can edit text files, that is nice. But what if you want just to view a text file?
Of course, you could use vim for that, but very often it is overkill. And two more thing
to consider: If you want to view really large file, you will want to view it in a program
which is as fast as possible.Often you would not want to accidentally change
something in a file.So I introduce you to the mighty less, which is more. “More than
what?” you might ask. Well… Once upon a time there was a viewer which was called
more. It was pretty simple, and just showed you the text file you asked it to show. It
was so simple that it could show you text files in only one direction, which is forward.
This was found unsatisfactory by Mark Nudelman, who wrote less it in 1983-85, and
since then it received many advanced features. Because it was more advanced than
more, a saying was born: “Less is more, and more is less”.
Well, let us try it.
Type this:
less .bashrc
You should see this:
student@vm1:~$ less .bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples
# If not running interactively, don't do anything
[ -z "$PS1" ] && return
# don't put duplicate lines in the history. See bash(1) for more options
# don't overwrite GNU Midnight Commander's setting of `ignorespace'.
HISTCONTROL=$HISTCONTROL${HISTCONTROL+:}ignoredups
.bashrc
If you don't have wide enough terminal, text will look like a mess because whole lines
will not fit in it. To fix it, type - -ch<ENTER><ENTER>. Yes, dash-dash-c-h-ENTERENTER. This will enable horizontal scrolling.
To navigate text up and down use already familiar j and k. To exit press q.
Now I will show you an advanced feature of less, which allows you see only those
lines you want. Type &enable<ENTER>. You should see this:
# enable color support of ls and also add hand
# enable programmable completion features (you
# this, if it's already enabled in /etc/bash.b
~
~
~
~
~
~
& (END)
Behold! To discard filter, just type &<ENTER>. Again, commands to remember:
j — move up.
k — move down.
q — quit less.
- -chop-long-lines or - -ch<ENTER><ENTER> — enable horizontal scrolling.
/ — search.
&something — show only those line from your file which contain something.
4
Extra credit
Linux has online manuals which are called by typing man. By default in out system
this manuals are viewed with less. Type man man and read it, then quit less.
That is it, not more extra credit for you.
5
Exercise 3. Bash: The shell, .profile,
.bashrc, .bash_history
When working with Linux using CLI (command line interface), you are interacting
with a program called a shell. All you type is passed to the shell, which interprets
what you have typed, does parameters expansion (which is somewhat similar to
brace expansion in algebra) and executes programs for you. Shell which we will be
using is called Bash, which stands for Bourne Again Shell, which is in turn a pun.
Now I will show you general overview of how the bash works in plain English:
YOU
Log into a Linux box
Your identity is verified by username (student) and password (Pa$$w0rd)
Bash is executed
Bash
reads and executes initial commands from your profile, which define:
What your command prompt will look like
What colors you will see when working in Linux
What you editor is
What you viewer is
...
After initial commands are read, Bash enters in a loop
While not ordered to exit by typing 'exit', 'logout' or hitting <CTRL+D>:
Read a line
Parse this line, expand braces
Execute command with expanded parameters
I repeat, any command which you type is not executed directly, but expanded firstly
and only then executed. For example, when you type ls * the star * is expanded into
list of all files in current directory.
Now you will learn how to modify your profile and how to write and view your history.
Do this
1: ls -al
2: cat .profile
3: echo Hello, $LOGNAME!
4: cp -v .profile .profile.bak
5: echo 'echo Hello, $LOGNAME!' >> .profile
6: tail -n 5 .profile
7: history -w
8: ls -altr
9: cat .bash_history
10: exit
What you should see
student@vm1's password:
Linux vm1 2.6.32-5-amd64 #1 SMP Sun May 6 04:00:17 UTC 2012 x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Jun 7 12:03:29 2012 from sis.site
Hello, student!
student@vm1:~$ ls -al
6
total 20
drwxr-xr-x 2 student student 4096 Jun 7 12:18 .
drwxr-xr-x 3 root root 4096 Jun 6 21:49 ..
-rw-r--r-- 1 student student 220 Jun 6 21:48 .bash_logout
-rw-r--r-- 1 student student 3184 Jun 6 21:48 .bashrc
-rw-r--r-- 1 student student 697 Jun 7 12:04 .profile
student@vm1:~$ cat .profile
# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login
# exists.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.
# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
#umask 022
# if running bash
if [ -n "$BASH_VERSION" ]; then
# include .bashrc if it exists
if [ -f "$HOME/.bashrc" ]; then
. "$HOME/.bashrc"
fi
fi
# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
PATH="$HOME/bin:$PATH"
fi
echo Hello, $LOGNAME!
student@vm1:~$ echo Hello, $LOGNAME!
Hello, student!
student@vm1:~$ cp -v .profile .profile.bak
`.profile' -> `.profile.bak'
student@vm1:~$ echo 'echo Hello, $LOGNAME!' >> .profile
student@vm1:~$ tail -n 5 .profile
# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
PATH="$HOME/bin:$PATH"
fi
echo Hello, $LOGNAME!
student@vm1:~$ history -w
student@vm1:~$ ls -altr
total 28
-rw-r--r-- 1 student student 3184 Jun 6 21:48 .bashrc
-rw-r--r-- 1 student student 220 Jun 6 21:48 .bash_logout
drwxr-xr-x 3 root root 4096 Jun 6 21:49 ..
-rw-r--r-- 1 student student 741 Jun 7 12:19 .profile.bak
-rw------- 1 student student 308 Jun 7 12:21 .bash_history
-rw-r--r-- 1 student student 697 Jun 7 12:25 .profile
drwxr-xr-x 2 student student 4096 Jun 7 12:25 .
student@vm1:~$ cat .bash_history
ls -al
cat .profile
echo Hello, $LOGNAME!
cp -v .profile .profile.bak
echo 'echo Hello, $LOGNAME!' >> .profile
7
tail -n 5 .profile
history -w
ls -altr
student@vm1:~$ exit
logout
Don not be afraid, all this commands will be explained. Line numbers correspond to
those “Now type this” section.
Explanation
Prints out all files in current directory, including hidden ones. Options -al tell ls to
print file list in long format and include all files including hidden ones. .profile and
.bash_rc are hidden files, because they start with dot .. Every file starting with dot is
hidden, it is that simple. This two particular files are shell scripts, They contain
instructions which are executed when you log in.
Prints out your .profile file. Just that.
Tells your shell, which is in your case bash, to output a string Hello, $LOGNAME!,
substituting $LOGNAME with environment variable $LOGNAME which happens to
contain your login.
Copies your .profile file to .profile.bak. Key -v tells cp to be verbose, which means
it will print out all operations. Remember this key, it is often used to tell commands to
feed you more information than they do by default.
Adds a line to your .bash_rc profile. From now on, every time you will log into vm1
this command will be executed. Attention, >> means to add something to file, but >
means replacing file with something. If you accidentally replaced your .profile
instead of appending to it, command
cp -v .profile.bak .profile
will return your old .profile file.
Prints out exactly five last lines from .profile file.
Writes all your command history to .bash_history file. Normally this is done at the
end of your session, when you close it by typing exit or by pressing <CTRL>+D.
Prints out files in current directory. Options -tr mean that file list is sorted by time in
reverse direction. This means that most recently created and modified files are
printed last. Notice how you have 2 new files now.
Prints out file which keeps history of your commands. Notice how all that you have
type is here.
Closes the session.
Extra credit
Search online why ls -al tells you “total 20” then there are only 5 files present. What
does it mean? Notice that ”.” and ”..” are special file entries, which correspond to the
current directory and parent directory respectively.
Log in vm1 and type man -K /etc/profile, now using cursor keys scroll to
“INVOCATION” section and read it. To exit man, type “q”. Type man man to find out
what -K man option means.
Type in uname with space before the command. Now, type history. You see? If you
prepend a command with space it won't be saved in history! Hint: usable when you
need to specify the password ring on the command line.
Find a wiki page about bash and try to read it. No worries if it scares you, just omit
frightening sections for now.
8
Exercise 4. Bash: working with files,
pwd, ls, cp, mv, rm, touch
In Linux everything is a file. But what is a file? For now it will suffice to say that it is
an object which contains some information. It is usually defined like this:
A computer file is a block of arbitrary information, or resource for storing information,
which is available to a computer program and is usually based on some kind of
durable storage. A file is durable in the sense that it remains available for programs
to use after the current program has finished. Computer files can be considered as
the modern counterpart of paper documents which traditionally are kept in offices'
and libraries' files, and this is the source of the term.
But this definition is too general, so let us be more specific. man stat tells us that file
is an object which has the following properties additionally to information it contains:
struct stat {
dev_t st_dev; /* ID of device containing file */
ino_t st_ino; /* inode number */
mode_t st_mode; /* protection */
nlink_t st_nlink; /* number of hard links */
uid_t st_uid; /* user ID of owner */
gid_t st_gid; /* group ID of owner */
dev_t st_rdev; /* device ID (if special file) */
off_t st_size; /* total size, in bytes */
blksize_t st_blksize; /* blocksize for file system I/O */
blkcnt_t st_blocks; /* number of 512B blocks allocated */
time_t st_atime; /* time of last access */
time_t st_mtime; /* time of last modification */
time_t st_ctime; /* time of last status change */
};
Don not be intimidated, just remember the following properties:
Size, which is exactly what is says on the tin.
Time of last access, which is updated when you view a file.
Time of last modification, which is updated when you change a file.
Now you will learn how to print out your current directory, files in your directory, copy
and move files.
Do this
1: pwd
2: ls
3: ls -a
4: ls -al
5: ls -altr
6: cp -v .bash_history{,1}
7: cp -v .bash_history1 .bash_history2
8: mv -v .bash_history1 .bash_history2
9: rm -v .bash_history2
10: touch .bashrc
11: ls -al
12: ls .*
What you should see
Hello, student!
student@vm1:~$ pwd
/home/student
student@vm1:~$ ls
9
student@vm1:~$ ls -a
. .. .bash_history .bash_history1 .bash_logout .bashrc .lesshst .profile .profile.bak .profile.bak1
student@vm1:~$ ls -al
total 40
drwxr-xr-x 2 student student 4096 Jun 7 13:30 .
drwxr-xr-x 3 root root 4096 Jun 6 21:49 ..
-rw------- 1 student student 853 Jun 7 15:03 .bash_history
-rw------- 1 student student 308 Jun 7 13:14 .bash_history1
-rw-r--r-- 1 student student 220 Jun 6 21:48 .bash_logout
-rw-r--r-- 1 student student 3184 Jun 6 21:48 .bashrc
-rw------- 1 student student 45 Jun 7 13:31 .lesshst
-rw-r--r-- 1 student student 697 Jun 7 12:25 .profile
-rw-r--r-- 1 student student 741 Jun 7 12:19 .profile.bak
-rw-r--r-- 1 student student 741 Jun 7 13:12 .profile.bak1
student@vm1:~$ ls -altr
total 40
-rw-r--r-- 1 student student 3184 Jun 6 21:48 .bashrc
-rw-r--r-- 1 student student 220 Jun 6 21:48 .bash_logout
drwxr-xr-x 3 root root 4096 Jun 6 21:49 ..
-rw-r--r-- 1 student student 741 Jun 7 12:19 .profile.bak
-rw-r--r-- 1 student student 697 Jun 7 12:25 .profile
-rw-r--r-- 1 student student 741 Jun 7 13:12 .profile.bak1
-rw------- 1 student student 308 Jun 7 13:14 .bash_history1
drwxr-xr-x 2 student student 4096 Jun 7 13:30 .
-rw------- 1 student student 45 Jun 7 13:31 .lesshst
-rw------- 1 student student 853 Jun 7 15:03 .bash_history
student@vm1:~$ cp -v .bash_history{,1}
`.bash_history' -> `.bash_history1'
student@vm1:~$ cp -v .bash_history1 .bash_history2
`.bash_history1' -> `.bash_history2'
student@vm1:~$ mv -v .bash_history1 .bash_history2
`.bash_history1' -> `.bash_history2'
student@vm1:~$ rm -v .bash_history2
removed `.bash_history2'
student@vm1:~$ touch .bashrc
student@vm1:~$ ls -al
total 36
drwxr-xr-x 2 student student 4096 Jun 14 12:23 .
drwxr-xr-x 3 root root 4096 Jun 6 21:49 ..
-rw------- 1 student student 853 Jun 7 15:03 .bash_history
-rw-r--r-- 1 student student 220 Jun 6 21:48 .bash_logout
-rw-r--r-- 1 student student 3184 Jun 14 12:24 .bashrc
-rw------- 1 student student 45 Jun 7 13:31 .lesshst
-rw-r--r-- 1 student student 697 Jun 7 12:25 .profile
-rw-r--r-- 1 student student 741 Jun 7 12:19 .profile.bak
-rw-r--r-- 1 student student 741 Jun 7 13:12 .profile.bak1
student@vm1:~$
student@vm1:~$ ls .*
.bash_history .bash_logout .bashrc .lesshst .profile .profile.bak .profile.bak1
.:
ls.out
..:
student
Explanation
10
Prints out your current working directory, which is your home directory. Notice how
this differs from ~ in student@vm1:~, which also indicates that your are in your
home directory. That is because ~ is a shortcut for your home directory.
You get nothing here because there are only hidden files in your home directory.
Remember, hidden files are ones which names start with a dot.
Prints out all files in your home directory, because -a parameter tells ls to show all
files including hidden ones.
Prints out all files in your home directory in long format: permissions, owner,
group, size, timestamp (normally modification time) and filename.
Notice how files are now arranged by date, newest files being last. -t tells ls to sort
by time, and -r tells ls to reverse the sort.
Copies .bash_history to .bash_history1. It may seem cryptic to you, but
explanation is really simple. Bash has a feature called brace expansion, that it has a
set of rules which define how constructs like something{1,2,3} are treated. In our
case .bash_history{,1} expands into two arguments, namely .bash_history and
.bash_history1. Bash just takes a parameter before the brace, .bash_history in our
case, and adds everything which is in braces, separated by commas, to this
parameter consequentially. First addition becomes just .bash_history because first
argument in braces is void, there is no first argument. Next, Bash adds 1 because
that's the second argument, and voila! Resulting arguments, which are passed to cp
after expansion are -v .bash_history .bash_history1.
This might be obvious for you now. Copies recently created .bash_history1 to
.bash_history2.
Moves .bash_history1 to .bash_history2. Notice that it overwrites the destination
file without asking, so there is no more .bash_history2, gone!
Removes .bash_history2. Where is no warning, so beware. Also, always use -v
key.
Updates .bashrc timestamp to current date and time. This means that all .bashrc
time attributes, st_atime, st_mtime and st_ctime are set to current date and time.
You may become sure of this by typing in stat .bashrc.
Prints out all files in long format again. Notice how .bashrc timestamps changed.
Prints out files in you home directory in short format. Notice that you get listing of not
only of /home/student directory, but also of /home directory itself. Do not use this
construct with any command, especially rm, ever! Chances are, you will accidentally
cripple your system by deleting wrong files or changes access right to them.
11
Exercise 5. Bash: environment
variables, env, set, export
Consider the following: you want a program to print out your username. How will this
program know it? For this in Linux there are environment variables. What this means
is that there are many variables in your shell, many of which are set up
automatically, and every time you run a program, some of them are passed to this
program.
To elaborate on this:
Some variables are set just for your current shell. They are called local shell
variables. You can list them by typing set, a bash built-in command, which means
that no other program is started then you execute it. This command is processed by
bash itself.
Other variables are passed to every program you launch from your current shell.
They are called environment variables and can be listed by env program, which
means that then typing env you will see what variables every program you start gets.
You may want do dig a bit deeper. To do this, grasp the concept of a subshell which
is a child process that is created when you run a program and which becomes your
program.
Now you will learn how to create variables and how to use them.
Do this
1: foo='Hello World!'
2: echo $foo
3: set | grep foo
4: env | grep foo
5: export foo
6: env | grep foo
7: foo='ls -al'
8: $foo
What you should see
student@vm1:~$ foo='Hello World!'
student@vm1:~$ echo $foo
Hello World!
student@vm1:~$ set | grep foo
foo='Hello World!'
student@vm1:~$ env | grep foo
student@vm1:~$ export foo
student@vm1:~$ env | grep foo
foo=Hello World!
student@vm1:~$ foo='ls -al'
student@vm1:~$ $foo
total 36
drwxr-xr-x 2 student student 4096 Jun 14 12:23 .
drwxr-xr-x 3 root root 4096 Jun 6 21:49 ..
-rw------- 1 student student 4042 Jun 15 18:52 .bash_history
-rw-r--r-- 1 student student 220 Jun 6 21:48 .bash_logout
-rw-r--r-- 1 student student 3184 Jun 14 12:24 .bashrc
-rw------- 1 student student 50 Jun 15 18:41 .lesshst
-rw-r--r-- 1 student student 697 Jun 7 12:25 .profile
-rw-r--r-- 1 student student 741 Jun 7 12:19 .profile.bak
12
-rw-r--r-- 1 student student 741 Jun 7 13:12 .profile.bak1
Explanation
Creates a variable foo and places the line “Hello World!” into it.
Prints out variable foo.
Prints out list of all your environment variables, which is passed to grep, which prints
out only lines containing variable foo. Notice how variable foo is present.
Prints out all list of your environment variables which are passed to any program you
execute. Notice how variable foo is absent.
Makes variable foo available to all programs you execute from the current shell.
Now you can see that your variable is indeed available to all programs you execute.
Puts ls / into variable foo.
Executes a command which is contained in variable foo.
Extra credit
Type in and execute env and set. See how many different variable there are? No
worries, usually you can quickly understand what a variable does by googling it. Try
to do this.
Try typing:
set | egrep '^[[:alpha:]].*$'
Now, try just set. See what I did here? Quick explanation is that
egrep '^[[:alpha:]].*$'
prints out only those lines which start with alphanumeric character, that is every
letter, and omits any other lines. Don't bother too much with this for now. Don not
bother about | symbol either, I will explain it later on.
Read about difference between env and set here:
http://stackoverflow.com/questions/3341372/differnce-between-the-shell-andenvironment-variable-in-bash Remember, stackoverflow is your friend! I will repeat it
many times.
Try typing set FOO=helloworld bash -c 'echo $FOO'. Explanation is here:
http://stackoverflow.com/questions/10938483/bash-specifying-environmentvariables-for-echo-on-command-line. Again, don not bother with this too much, it will
come to you after lots of practice automatically, I promise.
Try this:
PS1_BAK=$PS1
PS1='Hello, world! $(pwd) > '
PS1=$PS1_BAK
Notice how I used $(pwd) to access command output as a variable. Now, type man
bash, / (yes, just a slash), PS1, press <ENTER>. You can now press n to go to the
next result. Skim through PROMPTING and type q to exit man. Now, navigate to
http://stackexchange.com/ and search bash PS1. Understand how those two
documentation sources differ.
13
Exercise 6. Bash: language settings,
LANG, locale, dpkg-reconfigure locales
In Linux language selection is as simple as an exporting of a variable. That is right,
programs decide how to talk to you by looking into this varibale. Of course, for this to
work program must support locales and have translation to your language available
and installed. Let us see how it works by installing French locale.
Now you will learn how to install and select a locale.
Do this
1: echo $LANG
2: locale
3: man man # press q to exit man
4: sudo dpkg-reconfigure locales
Now, select the fr_FR.UTF-8 locale by navigating through the list using cursor keys
and select the locale using space. Select en_US.UTF-8 as the default system
locale.
5: export LANG=fr_FR.UTF-8
6: echo $LANG
7: locale # press q to exit man
8: man man
9: export LANG=en_US.UTF-8
What you should see
student@vm1:~$ echo $LANG
en_US.UTF-8
student@vm1:~$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
student@vm1:~$ man man
MAN(1)
Manual pager utils
MAN(1)
NAME
man - an interface to the on-line reference manuals
student@vm1:~$ sudo dpkg-reconfigure locales
---------------| Configuring locales |----------------------|
|
| Locales are a framework to switch between multiple
|
| languages and allow users to use their language,
|
| country, characters, collation order, etc.
|
|
|
14
| Please choose which locales to generate. UTF-8 locales |
| should be chosen by default, particularly for new
|
| installations. Other character sets may be useful for |
| backwards compatibility with older systems and software. |
|
|
|
<Ok>
|
|
|
-----------------------------------------------------------------------| Configuring locales |-------| Locales to be generated:
|
|
|
| [ ] fr_BE@euro ISO-8859-15
|
| [ ] fr_CA ISO-8859-1
|
| [ ] fr_CA.UTF-8 UTF-8
|
| [ ] fr_CH ISO-8859-1
|
| [ ] fr_CH.UTF-8 UTF-8
|
| [*] fr_FR ISO-8859-1
|
| [ ] fr_FR.UTF-8 UTF-8
|
| [ ] fr_FR@euro ISO-8859-15
|
|
|
|
|
|
<Ok>
<Cancel>
|
|
|
----------------------------------------------------------- Configuring locales ---------------------|
|
| Many packages in Debian use locales to display text in |
| the correct language for the user. You can choose a
|
| default locale for the system from the generated
|
| locales.
|
|
|
| This will select the default language for the entire |
| system. If this system is a multi-user system where not |
| all users are able to speak the default language, they |
| will experience difficulties.
|
|
|
|
<Ok>
|
|
|
------------------------------------------------------------------------ Configuring locales -------------| Default locale for the system environment: |
|
|
|
None
|
|
en_US.UTF-8
|
|
fr_FR.UTF-8
|
|
|
|
|
|
<Ok>
<Cancel>
|
|
|
----------------------------------------------Generating locales (this might take a while)...
en_US.UTF-8... done
fr_FR.UTF-8... done
Generation complete.
student@vm1:~$ export LANG=fr_FR.UTF-8
student@vm1:~$ echo $LANG
fr_FR.UTF-8
15
student@vm1:~$ locale
LANG=fr_FR.UTF-8
LANGUAGE=en_US:en
LC_CTYPE="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_MESSAGES="fr_FR.UTF-8"
LC_PAPER="fr_FR.UTF-8"
LC_NAME="fr_FR.UTF-8"
LC_ADDRESS="fr_FR.UTF-8"
LC_TELEPHONE="fr_FR.UTF-8"
LC_MEASUREMENT="fr_FR.UTF-8"
LC_IDENTIFICATION="fr_FR.UTF-8"
LC_ALL=
student@vm1:~$ man man
MAN(1) Utilitaires de l'afficheur des pages de manuel MAN(1)
NOM
man - interface de consultation des manuels de
référence en ligne
student@vm1:~$ export LANG=en_US.UTF-8
student@vm1:~$
Explanation
Prints out your current LANG variable which is used by programs to determine which
language to use when interacting with you.
Prints out all locale variables which are used by programs for setting up number,
address, phone format, etc. according to the format of the specified country.
Shows you manual page about unix manual system. Notice how I used # to
comment an action, everything after # is not executed.
Executes a program for re-configuring your locales. Because this change is system
wide, you need to run this command as root, that's why there is sudo in front of
dpkg-reconfigure locales. Don't bother about sudo now, I'd set it up for you.
Exports LANG variable which is used to set all other locale variables.
Prints out LANG variable, you can see that it's changed as you would expect.
Prints out other locale variables which are changed also.
Shows you man manual page, in French.
Reverts LANG variable back to English.
Extra credit
Read manual page about locale. To do this, type in man locale.
Now, read a man 7 locale page. Notice how I used 7 here to call a man page about
conventions. If you want, read man man now to understand what other possible
codes are, or just wait for exercise which covers it.
16
Exercise 7. Bash: redirection, stdin,
stdout, stderr, <, >, >>, |, tee, pv
In Linux, everything is just a file. This means that, for console programs:
Keyboard is represented as a file, from which Bash reads your input.
Display is represented as a file, to which Bash writes its output.
Let us say you have a program which counts lines in a file. You can call it by typing
wc -l. Try this now. Nothing happened, right? It just hangs there. Wrong, it is waiting
for your input. This is an outline of what it does:
line_counter = 0
while end of file is not reached
read a line
add 1 to line_counter
print line_counter
So wc currently reads lines from /dev/tty, which is in current context your keyboard.
A line is given to wc each time you hit Enter. Type any several lines, and then hit
CTRL+D, which will produce EOF character for wc to understand that end of file is
reached. Now it will tell you how much lines you entered.
But what if you want to count lines in existing file? Well, where is redirection for that!
To feed an existing file on its input, type this: wc -l < .bash_history. You see, it
works! And it really is that simple! redirection is a mechanism which allows you tell a
program to redirect its input and/or output from keyboard and display to another file.
To do this, you can use these special commands then starting a program:
< — replace standard input (for example keyboard) with a file.
> — replace standard output (for example display) with a file. Warning! This
command overwrites contents of file you specify, so be careful.
>> — same as above, but instead of overwriting a file it writes to the end of it,
preserving already existing information in this file. Be careful not to confuse the two.
| — Take an output from one program and connect it to the other. This will be
elaborated on in next exercise.
Now you will learn how to redirect input and output of programs to files or another
programs.
Do this
1: sudo aptitude install pv
2: read foo < /dev/tty
3: Hello World!
4: echo $foo > foo.out
5: cat foo.out
6: echo $foo >> foo.out
7: cat foo.out
8: echo > foo.out
9: cat foo.out
10: ls -al | grep foo
11: ls -al | tee ls.out
12: dd if=/dev/zero of=~/test.img bs=1M count=10
13: pv ~/test.img | dd if=/dev/stdin of=/dev/null bs=1
14: rm -v foo.out
15: rm -v test.img
What you should see
student@vm1:~$ sudo aptitude install pv
17
The following NEW packages will be installed:
pv
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/28.9 kB of archives. After unpacking 143 kB will be used.
Selecting previously deselected package pv.
(Reading database ... 39657 files and directories currently installed.)
Unpacking pv (from .../archives/pv_1.1.4-1_amd64.deb) ...
Processing triggers for man-db ...
Setting up pv (1.1.4-1) ...
student@vm1:~$ read foo < /dev/tty
Hello World!
student@vm1:~$ echo $foo > foo.out
student@vm1:~$ cat foo.out
Hello World!
student@vm1:~$ echo $foo >> foo.out
student@vm1:~$ cat foo.out
Hello World!
Hello World!
student@vm1:~$ echo > foo.out
student@vm1:~$ cat foo.out
student@vm1:~$ ls -al | grep foo
-rw-r--r-- 1 student student 1 Jun 15 20:03 foo.out
student@vm1:~$ ls -al | tee ls.out
total 44
drwxr-xr-x 2 student student 4096 Jun 15 20:01 .
drwxr-xr-x 3 root root 4096 Jun 6 21:49 ..
-rw------- 1 student student 4865 Jun 15 19:34 .bash_history
-rw-r--r-- 1 student student 220 Jun 6 21:48 .bash_logout
-rw-r--r-- 1 student student 3184 Jun 14 12:24 .bashrc
-rw-r--r-- 1 student student 1 Jun 15 20:03 foo.out
-rw------- 1 student student 50 Jun 15 18:41 .lesshst
-rw-r--r-- 1 student student 0 Jun 15 20:03 ls.out
-rw-r--r-- 1 student student 697 Jun 7 12:25 .profile
-rw-r--r-- 1 student student 741 Jun 7 12:19 .profile.bak
-rw-r--r-- 1 student student 741 Jun 7 13:12 .profile.bak1
-rw-r--r-- 1 student student 0 Jun 15 20:02 test.img
student@vm1:~$ dd if=/dev/zero of=~/test.img bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0130061 s, 806 MB/s
student@vm1:~$ pv ~/test.img | dd if=/dev/stdin of=/dev/null bs=1
10MB 0:00:03 [3.24MB/s]
[=================================================================================>] 100%
10485760+0 records in
10485760+0 records out
10485760 bytes (10 MB) copied, 3.10446 s, 3.4 MB/s
student@vm1:~$ rm -v foo.out
removed `foo.out'
student@vm1:~$ rm -v test.img
removed `test.img'
student@vm1:~$
Explanation
Install pv (pipe viewer) program on your system which you will need later on.
18
Reads your input to a variable foo. This is possible because display and keyboard
are actually a teletype for the system. Yes, that teletype! Read more about tty online.
This is what you type.
Redirects contents of the foo variable to the foo.out file, creating the file in the
process or overwriting an existing file without warning removing all there was!!
Prints out foo.out contents.
Redirects contents of the foo variable to the foo.out file, creating the file in the
process or appending to an existing file. This is safe, but don not confuse the two, or
great sorrow will fall upon you.
Prints out foo.out contents again.
Redirects nothing to the foo.out, emptying file in the process.
Shows that the file is, indeed, empty.
Lists your directory and pipes the output to grep. This works by taking all ls -al
output and feeding it into grep. Again, this is called piping.
Lists your directory to the screen and to the ls.out. Very useful!
Creates a zeroed file with size of 10 Megabytes. Don't bother with how this works for
now.
Reads this file to /dev/null, which is an ultimate garbage can in your system, a void.
Everything which is written into it goes nowhere. Notice how pv shows you a
progress of reading the file, if you try to read it with another command instead you
won't know how long it will take to complete.
Removes foo.out. Always remember to clean up after yourself.
Removes test.img.
Extra credit
Read about piping and redirection on stackoverflow, again on stackoverflow and on
Greg's Wiki which is very useful resource indeed, remember it.
Open man page for bash, scroll down to REDIRECTION section and read it.
Read descriptions from man pv and man tee.
19
Exercise 8. Bash: more on redirection
and filtering: head, tail, awk, grep, sed
Now that you tasted Linux, I will introduce you to the Unix way. Behold.
This is the Unix philosophy: Write programs that do one thing and do it well. Write
programs to work together. Write programs to handle text streams, because that is a
universal interface.
What this means in practical terms is that to become proficient in Linux you need to
know how to take an output from one program and feed it to the other, usually
modifying it in the process. Normally you do this by gluing several programs together
using pipes which allow you connect output of one program to another. Like this:
What happens here is really simple. Almost every Linux program opens at lest these
3 files when started:
stdin — the standard input. This is from where program reads something.
stdout — the standard output. This is where program writes something.
stderr — the standard error. This is where program whines about wrong things that
do happen.
This is how it reads:
Start Program 1
Start reading data from keyboard
Start writing errors to display
Start Program 2
Start reading input from Program 1
Start writing errors to Display
Start Program 3
Start reading input from Program 2
Start writing errors to Display
Start writing data to Display
There is another way to picture what happens if you like South Park type of humor,
but beware: what was seen cannot be unseen! Warning! You will not be able to
unsee this.
Let us consider the following pipeline which takes ls -al output and prints out only file
names and file modification times:
ls -al | tr -s ' ' | cut -d ' ' -f 8,9
20
Here is an outline of what happens:
Start ls -al
Get list of files in current directory
Write errors to Display
Write output to Pipe
Start tr -s ' '
Read input from ls -al via Pipe
Leave only 1 space between fields
Write errors to Display
Write output to Pipe
Start cut -d ' ' -f 8,9
Read input from tr -s ' ' via Pipe'
Leave only fields 8 and 9, throw away anything else
Write errors to Display
Write output to Display
To further elaborate, this is what happens on each step:
Step 1: ls -al, we get a directory listing. Every column here is called a field.
student@vm1:~$ ls -al
total 52
drwxr-xr-x 2 student student 4096 Jun 18 14:16 .
drwxr-xr-x 3 root root 4096 Jun 6 21:49 ..
-rw------- 1 student student 4865 Jun 15 19:34 .bash_history
-rw-r--r-- 1 student student 220 Jun 6 21:48 .bash_logout
-rw-r--r-- 1 student student 3184 Jun 14 12:24 .bashrc
-rw-r--r-- 1 student student 64 Jun 18 14:16 hello.txt
-rw------- 1 student student 89 Jun 18 16:26 .lesshst
-rw-r--r-- 1 student student 634 Jun 15 20:03 ls.out
-rw-r--r-- 1 student student 697 Jun 7 12:25 .profile
-rw-r--r-- 1 student student 741 Jun 7 12:19 .profile.bak
-rw-r--r-- 1 student student 741 Jun 7 13:12 .profile.bak1
-rw------- 1 student student 666 Jun 18 14:16 .viminfo
Step 2: ls -al | tr -s ' '. We leave only one space between fields because cut does
not understand multiple spaces as a way to tell several fields apart.
student@vm1:~$ ls -al | tr -s ' '
total 52
drwxr-xr-x 2 student student 4096 Jun 18 14:16 .
drwxr-xr-x 3 root root 4096 Jun 6 21:49 ..
-rw------- 1 student student 4865 Jun 15 19:34 .bash_history
-rw-r--r-- 1 student student 220 Jun 6 21:48 .bash_logout
-rw-r--r-- 1 student student 3184 Jun 14 12:24 .bashrc
-rw-r--r-- 1 student student 64 Jun 18 14:16 hello.txt
-rw------- 1 student student 89 Jun 18 16:26 .lesshst
-rw-r--r-- 1 student student 634 Jun 15 20:03 ls.out
-rw-r--r-- 1 student student 697 Jun 7 12:25 .profile
-rw-r--r-- 1 student student 741 Jun 7 12:19 .profile.bak
-rw-r--r-- 1 student student 741 Jun 7 13:12 .profile.bak1
-rw------- 1 student student 666 Jun 18 14:16 .viminfo
Step 3: ls -al | tr -s ' ' | cut -d ' ' -f 8,9. We leave only fields eight and nine, which are
what we want.
student@vm1:~$ ls -al | tr -s ' ' | cut -d ' ' -f 8,9
14:16 .
21:49 ..
19:34 .bash_history
21:48 .bash_logout
12:24 .bashrc
14:16 hello.txt
16:26 .lesshst
21
20:03 ls.out
12:25 .profile
12:19 .profile.bak
13:12 .profile.bak1
14:16 .viminfo
Now you will learn how to take output (text stream) from one program and pass it
another, and how transform it.
Do this
1: ls -al | head -n 5
2: ls -al | tail -n 5
3: ls -al | awk '{print $8, $9}'
4: ls -al | awk '{print $9, $8}'
5: ls -al | awk '{printf "%-20.20s %s\n",$9, $8}'
6: ls -al | grep bash
7: ls -al > ls.out
8: cat ls.out
9: cat ls.out | sed 's/bash/I replace this!!!/g'
What you should see
student@vm1:~$ ls -al | head -n 5
total 52
drwxr-xr-x 2 student student 4096 Jun 18 14:16 .
drwxr-xr-x 3 root root 4096 Jun 6 21:49 ..
-rw------- 1 student student 4865 Jun 15 19:34 .bash_history
-rw-r--r-- 1 student student 220 Jun 6 21:48 .bash_logout
student@vm1:~$ ls -al | tail -n 5
-rw-r--r-- 1 student student 636 Jun 18 17:52 ls.out
-rw-r--r-- 1 student student 697 Jun 7 12:25 .profile
-rw-r--r-- 1 student student 741 Jun 7 12:19 .profile.bak
-rw-r--r-- 1 student student 741 Jun 7 13:12 .profile.bak1
-rw------- 1 student student 666 Jun 18 14:16 .viminfo
student@vm1:~$ ls -al | awk '{print $8, $9}'
14:16 .
21:49 ..
19:34 .bash_history
21:48 .bash_logout
12:24 .bashrc
14:16 hello.txt
16:26 .lesshst
17:52 ls.out
12:25 .profile
12:19 .profile.bak
13:12 .profile.bak1
14:16 .viminfo
student@vm1:~$ ls -al | awk '{print $9, $8}'
. 14:16
.. 21:49
.bash_history 19:34
.bash_logout 21:48
.bashrc 12:24
hello.txt 14:16
.lesshst 16:26
ls.out 17:52
.profile 12:25
.profile.bak 12:19
22
.profile.bak1 13:12
.viminfo 14:16
student@vm1:~$ ls -al | awk '{printf "%-20.20s %s\n",$9, $8}'
.
14:16
..
21:49
.bash_history
19:34
.bash_logout
21:48
.bashrc
12:24
hello.txt
14:16
.lesshst
16:26
ls.out
17:52
.profile
12:25
.profile.bak
12:19
.profile.bak1
13:12
.viminfo
14:16
student@vm1:~$ ls -al | grep bash
-rw------- 1 student student 4865 Jun 15 19:34 .bash_history
-rw-r--r-- 1 student student 220 Jun 6 21:48 .bash_logout
-rw-r--r-- 1 student student 3184 Jun 14 12:24 .bashrc
student@vm1:~$ ls -al > ls.out
student@vm1:~$ cat ls.out
total 48
drwxr-xr-x 2 student student 4096 Jun 18 14:16 .
drwxr-xr-x 3 root root 4096 Jun 6 21:49 ..
-rw------- 1 student student 4865 Jun 15 19:34 .bash_history
-rw-r--r-- 1 student student 220 Jun 6 21:48 .bash_logout
-rw-r--r-- 1 student student 3184 Jun 14 12:24 .bashrc
-rw-r--r-- 1 student student 64 Jun 18 14:16 hello.txt
-rw------- 1 student student 89 Jun 18 16:26 .lesshst
-rw-r--r-- 1 student student 0 Jun 18 17:53 ls.out
-rw-r--r-- 1 student student 697 Jun 7 12:25 .profile
-rw-r--r-- 1 student student 741 Jun 7 12:19 .profile.bak
-rw-r--r-- 1 student student 741 Jun 7 13:12 .profile.bak1
-rw------- 1 student student 666 Jun 18 14:16 .viminfo
student@vm1:~$ cat ls.out | sed 's/bash/I replace this!!!/g'
total 48
drwxr-xr-x 2 student student 4096 Jun 18 14:16 .
drwxr-xr-x 3 root root 4096 Jun 6 21:49 ..
-rw------- 1 student student 4865 Jun 15 19:34 .I replace this!!!_history
-rw-r--r-- 1 student student 220 Jun 6 21:48 .I replace this!!!_logout
-rw-r--r-- 1 student student 3184 Jun 14 12:24 .I replace this!!!rc
-rw-r--r-- 1 student student 64 Jun 18 14:16 hello.txt
-rw------- 1 student student 89 Jun 18 16:26 .lesshst
-rw-r--r-- 1 student student 0 Jun 18 17:53 ls.out
-rw-r--r-- 1 student student 697 Jun 7 12:25 .profile
-rw-r--r-- 1 student student 741 Jun 7 12:19 .profile.bak
-rw-r--r-- 1 student student 741 Jun 7 13:12 .profile.bak1
-rw------- 1 student student 666 Jun 18 14:16 .viminfo
Explanation
Prints out only 5 first entries in directory listing.
Prints out only 5 last entries in directory listing.
Prints out only modification time and file name. Notice how I used awk, which is
smarter than cut. Difference here is that while cut understands only single symbol
23
(space in our case) as a way to tell field apart (field separator), awk treats any
number of spaces and tabs as filed separator, so there is no need to use tr which
removes unnecessary spaces.
Prints out file name and modification time in this order. This again is something cat is
not able to do.
Prints out file name and modification time nicely. Notice how output looks much
clearer now.
Prints out only those lines from directory listing which contain the word “bash”.
Writes directory listing output to file ls.out.
Prints out ls.out. cat is simplest program available that allows you to print out a file
and nothing more. Despite being so simple it is very useful when constructing
complicated pipelines.
Prints out ls.out replacing all bash entries with I replace this!!! sed is one powerful
stream editor, which is very very very useful.
Extra credit
Open man pages for head, tail, awk, grep and sed. Do not be intimidated, just
remember that man pages are always there for you. With some practice you will be
able actually understand them.
Find grep options which allow to print out one line before and one line after the lines
it finds.
Google about awk printf command, try to understand how this works.
Read Useless Use of Cat Award. Try some examples from there.
24
Exercise 9. Bash: job control, jobs, fg
Linux is a multitasking operating system. This means that there are many programs
running at the same time. From user perspective this means that you are able to run
several programs simultaneously, and bash surely has tools to control multiple jobs
execution for you. To be able to use this feature you need to learn the following
commands:
<CTRL>+z — place currently running program in the background.
jobs — list all background programs.
fg — bring a program to foreground. fg takes a number which can be obtained from
jobs as an argument, or brings last suspended program to foreground if called
without parameters.
<CTRL>+c — stop execution of currently running programs at once. While I will not
use it in this exercise, I must say that it may be very useful at times.
Now you will learn how to control program execution by using tools built in bash.
Do this
1: less -S .profile
2: <CTRL+z>
3: less -S .bashrc
4: <CTRL+z>
5: less -S .bash_history
6: <CTRL+z>
7: jobs
8: fg
9: q
10: fg
11: q
12: fg
13: q
14: fg
15: jobs
What you should see
student@vm1:~$ less -S .profile
# exists.
# see /usr/share/doc/bash/examples/startup-files for
# the files are located in the bash-doc package.
# the default umask is set in /etc/profile; for setti
# for ssh logins, install and configure the libpam-um
#umask 022
# if running bash
if [ -n "$BASH_VERSION" ]; then
# include .bashrc if it exists
if [ -f "$HOME/.bashrc" ]; then
. "$HOME/.bashrc"
[1]+ Stopped
less -S .profile
student@vm1:~$ less -S .bashrc
# for examples
# If not running interactively, don't do anything
25
[ -z "$PS1" ] && return
# don't put duplicate lines in the history. See bash(
# don't overwrite GNU Midnight Commander's setting of
HISTCONTROL=$HISTCONTROL${HISTCONTROL+:}ignoredups
# ... or force ignoredups and ignorespace
HISTCONTROL=ignoreboth
# append to the history file, don't overwrite it
shopt -s histappend
[2]+ Stopped
less -S .bashrc
student@vm1:~$ less -S .bash_history
echo Hello, $LOGNAME!
echo 'echo Hello, $LOGNAME!' >> .profile
cp .profile .profile.bak
tail .profile
ls -altr
history -w
ls -al
cat .profile
echo Hello, $LOGNAME!
echo 'echo Hello, $LOGNAME!' >> .profile
cp .profile .profile.bak
tail .profile
ls -altr
[3]+ Stopped
less -S .bash_history
student@vm1:~$ jobs
[1] Stopped
less -S .profile
[2]- Stopped
less -S .bashrc
[3]+ Stopped
less -S .bash_history
student@vm1:~$ fg
student@vm1:~$ fg
student@vm1:~$ fg
student@vm1:~$ fg
-bash: fg: current: no such job
student@vm1:~$ jobs
student@vm1:~$
Explanation
Opens .profile for viewing. Notice how I use -S parameter, which tells less to staring
with - -chop-long-lines option enabled.
Suspends less.
Opens .bashrc for viewing.
Suspends less.
Opens .bash_history for viewing.
Suspends less.
Prints out list of suspended programs.
Switches to last less.
Quits it.
Switches to second less.
Quits it.
Switches to first less.
Quits it.
26
Tries to switch to last program. There is no programs left, but you do this is to make
sure there are indeed none.
Prints out list of suspended programs. This is to make sure there are no background
jobs by seeing that jobs prints out empty output.
Extra credit
Open man bash, search for JOB CONTROL typing /, JOB CONTROL, <ENTER>,
and read it.
27
Exercise 10. Bash: program exit code
(return status)
Let us say that you want to copy a directory. You may do it by typing cp -vR
/old/dir/path /new/dir/path. After issuing this command you may want to know how
did it go. Was the directory copied? Or some error occurred due to the lack of space
in destination directory or something else that did go wrong?
To understand how it works you must understand how two programs may
communicate. Let us start with stating that bash is just another program, so
generally when you issued aforementioned cp command, one program (bash, which
is a parent process) called another (cp, which is child process).
In Linux there is a standard mechanism for getting information from child process to
parent process, and this mechanism is called exit status, or return code. By using
this mechanism a small number is passed from a child process (or callee, in our
case this is cp) to a parent process (or caller, which is bash in our case) when child
process finishes its work. When program encounters no errors during execution it
returns zero, 0. If some errors did occur, this code is non-zero. That simple. In Bash
this exit code is saved into ? environment variable, which as you know now can be
accessed as $?.
Let me reiterate what I said once more, now with outlines:
Bash waits for you input
Bash parses your input
Bash starts a program for you and waits for this program to exit
Program starts
Program does what you told it
Program generates the exit code
Program exits and returns this exit code to Bash
Bash sets this exit code to the variable ?
Now you will learn how to print out program exit status.
Do this
1: ls
2: echo $?
3: ls /no/such/dir
4: echo $?
What you should see
student@vm1:~$ ls
hello.txt ls.out
student@vm1:~$ echo $?
0
student@vm1:~$ ls /no/such/dir
ls: cannot access /no/such/dir: No such file or directory
student@vm1:~$ echo $?
2
student@vm1:~$
Explanation
Prints out a directory, successfully.
Prints out ls exit code, this is 0 which means that ls encountered no errors.
28
Tries to print out nonexistent directory, failing in the process of course.
Prints out ls /no/such/dir exit code, which is indeed non-zero.
Extra credit
Read Exit status section of man ls.
29
Exercise 11. Bash: wrapping up
Now that you have tasted how working in Linux using CLI feels, your next step is to open
your favorite text editor and make yourself the following table. Search those commands and
symbols meaning of which you do not know. Warning! For this to work you must type this
table by hand. Search for those terms and commands which are new to you.
Now you will learn how to grind something. And remember, no copy-pasting!
Terms
Term
Meaning
________________________________ ______________________________________
vim normal mode
vim command mode
CLI
Shell
Profile
File
File descriptor
Process
Program
Environment
Environment variable
Redirection
Pipe
Text stream
Stdin
Stdout
Stderr
EOF
Filtering
Job
Foreground job
Background job
Exit code
vim
Command
Meaning
_____________________________
______________________________________
vim
h
j
k
l
i
o
30
<ESCAPE>
x
dd
:wq
:q!
/
less
Command
_____________________________
less
j
k
q
--ch
/
&
Bash and Bash built-in commands
Command
_____________________________
echo
history
exit
pwd
=
$
?
set
env
export
$LANG
read
<CTRL>+z
<CTRL>+c
jobs
fg
Redirection
Command
_____________________________
>
<
>>
|
/dev/stdin
/dev/stdout
/dev/stderr
Meaning
______________________________________
Meaning
______________________________________
Meaning
______________________________________
31
Other programs you have learnt
Command
Meaning
_____________________________
______________________________________
man
ls
cat
dpkg-reconfigure
head
tail
grep
awk
sed
tee
dd
pv
locale
sudo
cp
mv
rm
touch
wc
After filling in the table, write every command on a post-it note and repeat before going to
sleep for a week. Yes, I mean it, shake dust from those pen and paper and do it.
32
Exercise 12. Documentation: man, info
Now that you have tasted Linux it is time to introduce online Linux documentation
facilities. You already know about man, because I told you to look things up in it.
Maybe you even had read man documentation page. So anyway, what do you need
to know about man to use it effectively?
To begin with, man pages are just compressed text files containing special markup
so man program will know how to format them for you. In Debian they are located in
/usr/share/man/. You can browse them by using zless. It is not even a program but
a shell script, which uncompresses files and invokes less.
Next, I will quote man page about its sections:
Executable programs or shell commands
System calls (functions provided by the kernel)
Library calls (functions within program libraries)
Special files (usually found in /dev)
File formats and conventions eg /etc/passwd
Games
Miscellaneous (including macro packages and conventions), e.g. man(7), groff(7)
System administration commands (usually only for root)
Kernel routines [Non standard]
It is exactly what it says on the tin. To invoke appropriate man section just type its
section number, like: man 1. No worries if you do not understand what some section
means, for now you will need only sections 1 and 8, which are about programs and
system administrator utilities installed on your system. Additionally, you man already
know what man(7) is for.
This are standard sections of a man page:
NAME — program name and short description.
SYNOPSIS — short list of available program options
DESCRIPTION — description of a program and explanation of available parameters.
OPTIONS — some man pages keep explanation of available parameters here.
EXIT STATUS — every program returns a code which indicates its success or
failure. This code values are explained here.
RETURN VALUE — usually the same as EXIT STATUS.
ERRORS — known errors in a program.
ENVIRONMENT — environment variables. Set those before invoking the program.
FILES — usually, program configuration files.
VERSIONS — information about changes in program.
CONFORMING TO — notes about compatibility.
NOTES — information which an author of a man page did not know where to put.
BUGS — known errors in a program.
EXAMPLE — contains examples of program invocation. Very useful!
AUTHORS — who wrote the program.
SEE ALSO — related man pages.
Now about conventions, quoting again:
bold text — type exactly as shown.
italic text — replace with appropriate argument. This text mostly is displayed not
italic, but like underlined.
[-abc] — any or all arguments within [ ] are optional.
-a|-b — options delimited by | cannot be used together.
33
argument … — argument is repeatable.
[expression] … — entire expression within [ ] is repeatable.
I will illustrate this by example. man less shows this:
Well, looks kinda scary I must say. First 4 lines are easy, just type what is shown and
that is it:
1. less -?
2. less –help
3. less -V
4. less –version
Starting from line 5, as we can see, italic text is indeed shown as underlined. And, it
looks totally incomprehensible. Let us go through it line by line.
5. less [-[+]aBcCdeEfFgGiIJKLmMnNqQrRsSuUVwWX~] — this looks more
scary.
First, it is optional because all arguments are enclosed in [].
Second, when specifying an argument it must start with -. This is non-optional.
Third, after - you may specify optional modifier +, which meaning is explained further
in the manual.
Fourth, you can specify one or several of commands which are shown as sequence
of letters here. For example, you may type less -S .bashrc, or less -+S .bashrc, or
less -SG .bashrc .profile or less -+SG .bashrc .profile
6. [-b space] [-h lines] [-j line] [-k keyfile] — simple enough, you may specify any
of the keys shown -b, -h, -j, -k with parameters space, lines, line and keyfile
respectively, which are explained in further in the manual.
7. [-{oO} logfile] [-p pattern] [-P prompt] [-t tag] — almost the same as line 6. {oO} means that you may specify -o or -O, but not both at the same time.
8. [-T tagsfile] [-x tab,…] [-y lines] [-[z] lines] — again, almost the same as line 6. x tab,… means that you may specify several values after -x, for example -x9 or x9,17. -[z] lines means, that -z is optional, and you may just type less -10 instead of
less -z10.
9. [-# shift] [+[+]cmd] [- -] [filename]… — this is somewhat more cryptic. +[+]cmd
means that you may type less +cmd or less ++ cmd. - - is just a prefix.
34
[filename]… reads one-or-more and means that you may specify several files when
invoking less, for example less .bashrc, less .bashrc .profile, etc.
We are done! Not so scary, is it? Remember, because you are viewing manuals with
less, to search what some key means type /key<ENTER> or &key<ENTER>. For
example, to search what -T key means, type /-T<ENTER>.
Now I will give you list of useful man commands:
man -k . — list all man pages in your system. Not exactly very useful, but you may
want to mediate on this list sometime. Or you can just count them all by typing man k . | wc.
man -k [search string] — search something in man pages descriptions. Tye this:
man -k tty.
man -wK [search string] - search something in man page bodies. Try man -wK tty.
Well, that is it for man. Now, there is another useful documentation utility, info. The
command list is as follows:
info […] — invoke info. If you invoke it without arguments, it will take you to the
index page.
<UP>, <DOWN>, <LEFT>, <RIGHT> allows you to scroll text.
<ENTER> open the link under cursor. Links start with * .
<TAB> — jump to the next link in document.
u — go up one level.
p — go to previous page, just like in browser.
n — go to next pages.
q — close info.
To start info with already familiar to you as I hope, vi keys, type info –vi-keys. Now
you can scroll with j and k.
Extra credit
Type man man and try to decode SYNOPSIS section, which explains how it may be
invoked.
Type info and h, read info help section.
35
Exercise 13. Documentation: Google
Introduction to finding documentation
Now that you know how to work with online Linux documentation, I will tell you this:
“Online Linux documentation is good, but it just is not enough.” This means that man
pages are useful if you already familiar with how a particular program works, but are
very unhelpful when you do not.
To get yourself started you need to read a book or find a small recipe which allows
you get started which is called a “how to”. For example, to get started with Apache
web-server you may google for “apache how to”. That is ok, and that is what Google
is for, but now I will give you a big warning:
Do not follow any How To blindly, ever!
The correct way to use Google is this:
Find a “How To”.
Follow it, but read, or at least skim, through all man pages for programs about which
you do not know about. Also, read about all options in the howto which are unknown
to you. It is that important.
Not paste, but type most of commands, the hard way. It requires some effort, but
really helps to memorize them and even better understand what you are doing.
List of useful resources
Sometimes it is better to search a particular site instead of blindly typing stuff into
Google. This is list of the useful resources:
http://en.wikipedia.org is invaluable when getting initial information about some topic.
Its links section is even more invaluable.
http://stackexchange.com/ This is very useful site for finding information about usage
examples and use cases. StackExchange network includes several resources, of
which most useful to you are http://serverfault.com/ and
http://unix.stackexchange.com/. http://stackoverflow.com/ is a very useful resource
when you are doing bash scripting.
http://www.cyberciti.biz/ contains many useful “how to's” and examples.
Homepages of many programs provide good and sometimes great documentation.
For example this are for apache and ngnix, respectively:
http://httpd.apache.org/docs/, http://nginx.org/en/docs/.
http://tldp.org/ is The Linux Documentation Project and contains many in-depth
guides about different topics.
Search tips
Google has a query language which gives you ability to execute powerful queries.
This is main commands of this language:
(screen|tmux) how to — search for screen and tmux howtos at the same time.
Remember about shell parameter expansion? This is similar.
site:serverfault.com query — search something on this site only. You may search
several sites at once with (site:serverfault.com | site:stackexchange.com)
“a long query” — shows you only those pages which contain this query exactly.
-query — excludes something from search results.
Extra credit
36
Go to http://www.google.com/advanced_search and find out what else can be
searched. Try searching for file types and last updated pages.
37
Exercise 14. Package management:
Debian package management utility
aptitude
Now it is time to get the sacred knowledge of adding new programs to your Linux
system. The programs in Linux are called packages, and are usually installed from
network repositories by tools knows as package managers.
Package is typically a compressed program, and you install a package just like this:
aptitude install program . . . To avoid installation of malicious programs, all
packages are digitally signed by their creators, which means that if package is
modified after its creation package manager will not allow you to install it.
Package manager is a program which allows you install other program. Many
programs depend on other programs, for example a program which uses dialog
windows typically needs a program which knows how to draw those windows.
Package manager knows about this dependencies, and when you ask it to install a
particular programs it installs all programs which are needed for the one you asked
to work. Debian package manager is called aptitude.
Network repository is a site with many packages, readily available for installation.
This is an outline of typical program installation:
YOU
Search for available programs using package manager
Ask your package manager to install a program
Package manager
Looks up for all programs which are needed to install current one
Marks them for installation in package manager database
Installs all needed programs including the one you need
Downloads all needed programs
Extracts files from those packages to places on your system defined by FHS standard
For every program runs a special install script which allow it to perform initial actions:
Create a directory
Create a database
Generate default configuration file
...
Updates the system package database by modifying status of installed programs to installed
YOU
Are able to run you freshly installed program now
It is time to understand where extracted files go. In Linux all same types of files are
installed in the same locations. For example, executable files from all programs are
installed in /usr/bin, documentation from all programs in /usr/share/doc and so on.
This may sound a bit messy, but it is very useful. Which files go where is defined a
document called FHS standard, you can view it by invoking man 7 hier. I will show
you abbreviated version of The Filesystem Hierarchy Standard, Version 2.2 below:
/ – This is the root directory. This is where the whole tree starts.
/bin – This directory contains executable programs which are needed in single user
mode and to bring the up or repair it.
/boot – Contains static files for the boot loader. This directory only holds the files
which are needed the boot process. The map installer and configuration files should
go to /sbin and /etc.
/dev – Special or device files, which refer to physical devices. See mknod(1).
/etc – Contains configuration files which are local to the machine.
38
/home – On machines with home directories for users, these are usually beneath
this directory, directly o The structure of this directory depends on local
administration decisions.
/lib – This directory should hold those shared libraries that are necessary to boot the
system and to run commands in the root file system.
/media – This directory contains mount points for removable media such as CD and
DVD disks or USB sticks.
/mnt – This directory is a mount point for a temporarily mounted file system. In some
distributions, /mnt contains subdirectories intended to be used as mount points for
several temporary file systems.
/proc – This is a mount point for the proc file system, which provides information
about running processes and the kernel. This pseudo-file system is described in
more detail in proc(5).
/root – This directory is usually the home directory for the root user (optional).
/sbin – Like /bin, this directory holds commands needed to boot the system, but
which are usually not executed by normal users.
/srv – This directory contains site-specific data that is served by this system.
/tmp – This directory contains temporary files which may be deleted with no notice,
such as by a regular job or at system boot up.
/usr – This directory is usually mounted from a separate partition. It should hold only
sharable, read-only data, so that it can be mounted by various machines running
Linux.
/usr/bin – This is the primary directory for executable programs. Most programs
executed by normal users which are not needed for booting or for repairing the
system and which are not installed locally should be placed in this directory.
/usr/local – This is where programs which are local to the site typically go.
/usr/share – This directory contains subdirectories with specific application data, that
can be shared among different architectures of the same OS. Often one finds stuff
here that used to live in /usr/doc or /usr/lib or /usr/man.
/usr/share/doc – Documentation about installed programs.
/var – This directory contains files which may change in size, such as spool and log
files.
/var/log – Miscellaneous log files.
/var/spool – Spooled (or queued) files for various programs.
/var/tmp – Like /tmp, this directory holds temporary files stored for an unspecified
duration.
Well, that was big. But you do not need to memorize it, man hier 7 is always there
for you. For now you need to know only about /usr/bin, /usr/share and /var/log.
Let us talk about packages and package managers a little more. First, let us
reiterate:
Every program is called a package.
Package manager manages all you packages, that is installs and uninstalls them for
you.
To be able to do this, package managers holds a database of installed and available
packages.
Every package in this database has a status, which indicates whether package is
installed, if it can be updated and so on. You can print out currently installed
packages by typing dpkg -l. Example output is shown below:
student@vm1:~$ dpkg -l | head | less -S
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
39
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name
Version
Description
+++-=====================-============================================================================
ii acpi
1.5-2
displays information on ACPI devices
ii acpi-support-base 0.137-5
scripts for handling base ACPI events such as the power
ii acpid
1:2.0.7-1squeeze4 Advanced Configuration and Power Interface event daemon
As you can see, those statuses are shown in first three columns. From this output we
can tall that all package are desired to be installed, are indeed installed and there is
no error because third column is empty. Here is a list of all possible package
statuses.
First column. Desired action, what we want to do with package:
u = Unknown (an unknown state)
i = Install. The package is selected for installation.
r = The package is selected for deinstallation (i.e. we want to remove all files, except
configuration files).
p = Purge. The package is selected to be purged (i.e. we want to remove everything
from system directories, even configuration files).
h = A package marked to be on hold is not handled by dpkg, unless forced to do that
with option –force-hold.
Second column. Package status, in what state package currently is:
n = Not-installed. The package is not installed on your system.
c = Config-files. Only the configuration files of the package exist on the system.
H = Half-installed. The installation of the package has been started, but not
completed for some reason.
U = Unpacked. The package is unpacked, but not configured.
F = Half-configured. The package is unpacked and configuration has been started,
but not yet completed for some reason.
W = Triggers-awaiting. The package awaits trigger processing by another package.
t = Triggers-pending. The package has been triggered.
i = Installed. The package is unpacked and configured OK.
Third column. Something got wrong:
R = reinst-required. A package marked reinst-required is broken and requires
reinstallation. These packages cannot be removed, unless forced with option –forceremove-reinstreq.
Again, you don not need to memorize this, just remember info dpkg command
which will show you exactly this information. And do not bother with package
statuses for now, just remember that ii status means everything is ok with the
package.
Enough talking, let us install a program called midnight commander which is a file
manager, that is it allows you to visually navigate directories on your system and
perform such operations on your files as copying, renaming or deleting them.
Now you will learn how search, install and remove packages.
Do this
1: aptitude search mc | grep -i 'midnight commander'
2: sudo aptitude install mc
3: dpkg -L mc | grep '/usr/bin'
4: aptitude search mc | grep -i 'midnight commander'
5: mc
6: <F10><ENTER>
7: sudo aptitude remove mc
40
What you should see
student@vm1:~$ aptitude search mc | grep -i 'midnight commander'
p mc
- Midnight Commander - a powerful file manag
p mc-dbg
- Midnight Commander - a powerful file manag
student@vm1:/home/student# sudo aptitude install mc
The following NEW packages will be installed:
libglib2.0-0{a} libglib2.0-data{a} mc shared-mime-info{a}
0 packages upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,957 kB/5,157 kB of archives. After unpacking 17.0 MB will be used.
Do you want to continue? [Y/n/?] y
Get:1 http://mirror.yandex.ru/debian/ squeeze/main libglib2.0-0 amd64 2.24.2-1 [1,122 kB]
Get:2 http://mirror.yandex.ru/debian/ squeeze/main libglib2.0-data all 2.24.2-1 [994 kB]
Get:3 http://mirror.yandex.ru/debian/ squeeze/main shared-mime-info amd64 0.71-4 [841 kB]
Fetched 2,957 kB in 0s (4,010 kB/s)
Selecting previously deselected package libglib2.0-0.
(Reading database ... 24220 files and directories currently installed.)
Unpacking libglib2.0-0 (from .../libglib2.0-0_2.24.2-1_amd64.deb) ...
Selecting previously deselected package libglib2.0-data.
Unpacking libglib2.0-data (from .../libglib2.0-data_2.24.2-1_all.deb) ...
Selecting previously deselected package mc.
Unpacking mc (from .../mc_3%3a4.7.0.9-1_amd64.deb) ...
Selecting previously deselected package shared-mime-info.
Unpacking shared-mime-info (from .../shared-mime-info_0.71-4_amd64.deb) ...
Processing triggers for man-db ...
Setting up libglib2.0-0 (2.24.2-1) ...
Setting up libglib2.0-data (2.24.2-1) ...
Setting up mc (3:4.7.0.9-1) ...
Setting up shared-mime-info (0.71-4) ...
student@vm1:~$ aptitude search mc | grep -i 'midnight commander'
i mc
- Midnight Commander - a powerful file manag
p mc-dbg
- Midnight Commander - a powerful file manag
student@vm1:~$ mc
Left File Command Options Right
|< ~ ---------------------.[^]>||< ~ ---------------------.[^]>|
|'n Name | Size |Modify time||'n Name | Size |Modify time|
|/..
|P--DIR|un 6 21:49||/..
|P--DIR|un 6 21:49|
|/.aptitude | 4096|un 25 18:34||/.aptitude | 4096|un 25 18:34|
|/.mc
| 4096|un 25 18:41||/.mc
| 4096|un 25 18:41|
| .bash~story| 10149|un 21 12:01|| .bash~story| 10149|un 21 12:01|
| .bash~ogout| 220|un 6 21:48|| .bash~ogout| 220|un 6 21:48|
| .bashrc | 3184|un 14 12:24|| .bashrc | 3184|un 14 12:24|
| .lesshst | 157|un 25 11:31|| .lesshst | 157|un 25 11:31|
|----------------------------------------------------------------|
|UP--DIR
--UP--DIR
|
----------- 6367M/7508M (84%) -------------- 6367M/7508M (84%) -|
Hint: The homepage of GNU Midnight Commander: http://www.midnightstudent@vm1:~$
[^]
1Help 2Menu 3View 4Edit 5Copy 6Re~ov 7Mkdir 8De~te 9Pu~Dn
student@vm1:~$ sudo aptitude remove mc
The following packages will be REMOVED:
libglib2.0-0{u} libglib2.0-data{u} mc shared-mime-info{u}
0 packages upgraded, 0 newly installed, 4 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 17.0 MB will be freed.
Do you want to continue? [Y/n/?] y
(Reading database ... 24637 files and directories currently installed.)
41
Removing shared-mime-info ...
Removing mc ...
Removing libglib2.0-data ...
Removing libglib2.0-0 ...
Processing triggers for man-db ...
student@vm1:~$
Explanation
Search packages which name contains mc and show only whose packages which
contain “midnight commander” in description. grep -i means that grep should search
both lowercase and uppercase letters, without it grep will not show you lines
containing Midnight Commander because they start from uppercase letters. Notice
that mc has p status, which means that the desired action with this package is
purge, and because there is nothing in other two status columns, we can conclude
that this package is not installed. You man notice, that you did not install this
package to begin with, but it is ok, because not installed packages have purged
status by default.
Installs package “mc”. Because this change is system-wide this command needs to
be invoked as superuser, which is allowed to write to all directories in the sytem.
Also notice how debian package manager aptitude automatically installs packages
libglib2.0-0, libglib2.0-data and shared-mime-info which are needed for mc.
Shows you installed package executable files. As you can see, they go into /usr/bin.
Invokes mc.
Exits mc.
Removes mc. Notice that automatically installed packages are also removed
automatically. If after mc installation you would install something that needs those
packages, aptitude will keep them.
Extra creadit
Well, that was a big one. And here is more:
Type aptiutde search emacs. Find out what v means.
Read, or at list skim, through Chapter 2. Debian package management of Debian
manual.
42
Exercise 15. System boot: runlevels,
/etc/init.d, rcconf, update-rc.d
Firstly I will give an outline of typical system boot process:
YOU
Press power switch (or start a virtual machine)
Now computer has the control
This control is passed to BIOS
BIOS
Performs hardware-specific tasks
Makes POST, Power-On Self-Test, which tests your hardware
Detects installed hardware, such as hard disks, memory type and amount, ...
Initializes hardware by writing initial values into their memory
Finds a boot device, which is usually a hard disk
Read and executes MBR (Main Boot Record) located at the beginning of this disk
Control is not passed to MBR
MBR
MBR finds and executes GRUB (Grand Unified Bootloader)
Control is now passed to GRUB
GRUB
Finds available filesystems
Finds and reads its configuration file, to find out:
Where system is located
What system to boot
What other actions perform
Executes Linux Kernel, main part of Linux OS
Control is now passed to Linux Kernel
Linux Kernel
Finds and loads initrd, which is initial ram disk
initrd contains necessary drivers which allow to access and mount real filesystems
Mounts the root file system, which is specified in GRUB config file
Executes /sbin/init, a special program which starts all the others
Control is now passed to init
init
Looks at the /etc/inittab to determine desired run level
Loads all programs appropriate for this runlevel
All programs from /etc/rc.d/rc2.d/ are loaded, because 2 is default Debian runlevel
SSH and TTY are started so you are able to connect to your computer
Booting up is now finished
YOU
Connect to your computer using SSH
SSH daemon executes bash shell for you
You can now type stuff
You are in control once again
For now we are interested only in “init” and “runlevel” stages, so I will give you a
general overview of how the system starts up and automatically starts some
programs. First, a little terminology:
Daemon — a program which runs in background all the time. This means that it does
not care if you are logged into the system, and usually you do not need to start it
manually, because daemond are started automatically when computer boots up.
Runlevel — a mode of system operation. Basically, this is just numbers which are
fed to init program, which knows which daemons are associated with each number,
and startd and stops those daemons as needed.
In Debian there are the following runlevels:
43
ID Description
S This executed then system is powered on.
0 Halt, this defines which actions are executed when system is shutting down.
Single-User mode, this is a special mode for troubleshooting. In this mode most
1
daemons are not automatically started.
2Full Multi-User, configured daemon start in this mode.
5
6 Reboot. Like halt, but instead of powering down system reboots.
But how does init know? Well, where are special directories for this:
student@vm1:/etc$ find /etc -type d -name 'rc*' 2>/dev/null | sort
/etc/rc0.d
/etc/rc1.d
/etc/rc2.d
/etc/rc3.d
/etc/rc4.d
/etc/rc5.d
/etc/rc6.d
/etc/rcS.d
As you might guess, each number and S corresponds to a run level from this table.
Let us list one of those directories, which start all needed daemons on normal boot:
student@vm1:/etc$ ls -al /etc/rc2.d | awk '{printf "%-15.15s %-3.3s %s\n",$9,$10,$11}'
.
..
README
S14portmap
-> ../init.d/portmap
S15nfs-common -> ../init.d/nfs-common
S17rsyslog
-> ../init.d/rsyslog
S17sudo
-> ../init.d/sudo
S18acpid
-> ../init.d/acpid
S18atd
-> ../init.d/atd
S18cron
-> ../init.d/cron
S18exim4
-> ../init.d/exim4
S18ssh
-> ../init.d/ssh
S20bootlogs -> ../init.d/bootlogs
S21rc.local -> ../init.d/rc.local
S21rmnologin -> ../init.d/rmnologin
S21stop-bootlog -> ../init.d/stop-bootlogd
As you can see, files in this directory are just symbolic links to actual startup scripts.
Let us look at one of this link more closely: S18ssh → ../init.d/ssh. Here is what we
are able to say about this file:
It is a link to the file ./init.d/ssh
It starts with S which means start. Each script used in Debian start-up system has at
least 2 parameters, start and stop. Now we can say that when our system switches
to runlevel 2 this script is executed with action start.
It has a number 18. Scripts in rc directories are executed in alphabetical order, so
now we understand that before starting ssh, system starts portmap, nfs-common,
rsyslog, and sudo. rsyslog is a system loggin daemon, and ssh in particular want
to log who and when accessed the system, so it needs rsyslog to be running before
starting up.
Now you will learn how to list enabled services (daemons), and enable and disable
services (daemons).
Do this
1: sudo aptitude install rcconf
44
2: ls -al /etc/rc2.d
3: sudo rcconf --list
4: sudo update-rc.d exim4 disable
5: ls -al /etc/rc2.d
6: sudo rcconf --list
7: sudo update-rc.d exim4 enable
8: ls -al /etc/rc2.d
9: sudo rcconf --list
What you should see
student@vm1:/var/log$ sudo aptitude install rcconf
The following NEW packages will be installed:
rcconf
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/23.9 kB of archives. After unpacking 135 kB will be used.
Selecting previously deselected package rcconf.
(Reading database ... 24239 files and directories currently installed.)
Unpacking rcconf (from .../archives/rcconf_2.5_all.deb) ...
Processing triggers for man-db ...
Setting up rcconf (2.5) ...
student@vm1:/etc$ ls -al /etc/rc2.d
total 12
drwxr-xr-x 2 root root 4096 Jun 27 11:42 .
drwxr-xr-x 68 root root 4096 Jun 25 18:43 ..
-rw-r--r-- 1 root root 677 Mar 27 05:50 README
lrwxrwxrwx 1 root root 17 Jun 4 11:53 S14portmap -> ../init.d/portmap
lrwxrwxrwx 1 root root 20 Jun 4 11:53 S15nfs-common -> ../init.d/nfs-common
lrwxrwxrwx 1 root root 17 Jun 4 11:53 S17rsyslog -> ../init.d/rsyslog
lrwxrwxrwx 1 root root 14 Jun 15 19:02 S17sudo -> ../init.d/sudo
lrwxrwxrwx 1 root root 15 Jun 4 11:53 S18acpid -> ../init.d/acpid
lrwxrwxrwx 1 root root 13 Jun 4 11:53 S18atd -> ../init.d/atd
lrwxrwxrwx 1 root root 14 Jun 4 11:53 S18cron -> ../init.d/cron
lrwxrwxrwx 1 root root 15 Jun 27 11:42 S18exim4 -> ../init.d/exim4
lrwxrwxrwx 1 root root 13 Jun 4 11:53 S18ssh -> ../init.d/ssh
lrwxrwxrwx 1 root root 18 Jun 4 11:53 S20bootlogs -> ../init.d/bootlogs
lrwxrwxrwx 1 root root 18 Jun 4 11:53 S21rc.local -> ../init.d/rc.local
lrwxrwxrwx 1 root root 19 Jun 4 11:53 S21rmnologin -> ../init.d/rmnologin
lrwxrwxrwx 1 root root 23 Jun 4 11:53 S21stop-bootlogd -> ../init.d/stop-bootlogd
student@vm1:/etc$ sudo rcconf --list
rsyslog on
ssh on
bootlogs on
portmap on
sudo on
nfs-common on
udev on
console-setup on
kbd on
exim4 on
keyboard-setup on
acpid on
cron on
atd on
procps on
module-init-tools on
student@vm1:/etc$ sudo update-rc.d exim4 disable
45
update-rc.d: using dependency based boot sequencing
insserv: warning: current start runlevel(s) (empty) of script `exim4' overwrites defaults (2 3 4 5).
insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script `exim4' overwrites defaults (0 1 6).
student@vm1:/etc$ ls -al /etc/rc2.d
total 12
drwxr-xr-x 2 root root 4096 Jun 27 11:43 .
drwxr-xr-x 68 root root 4096 Jun 25 18:43 ..
lrwxrwxrwx 1 root root 15 Jun 27 11:43 K01exim4 -> ../init.d/exim4
-rw-r--r-- 1 root root 677 Mar 27 05:50 README
lrwxrwxrwx 1 root root 17 Jun 4 11:53 S14portmap -> ../init.d/portmap
lrwxrwxrwx 1 root root 20 Jun 4 11:53 S15nfs-common -> ../init.d/nfs-common
lrwxrwxrwx 1 root root 17 Jun 4 11:53 S17rsyslog -> ../init.d/rsyslog
lrwxrwxrwx 1 root root 14 Jun 15 19:02 S17sudo -> ../init.d/sudo
lrwxrwxrwx 1 root root 15 Jun 4 11:53 S18acpid -> ../init.d/acpid
lrwxrwxrwx 1 root root 13 Jun 4 11:53 S18atd -> ../init.d/atd
lrwxrwxrwx 1 root root 14 Jun 4 11:53 S18cron -> ../init.d/cron
lrwxrwxrwx 1 root root 13 Jun 4 11:53 S18ssh -> ../init.d/ssh
lrwxrwxrwx 1 root root 18 Jun 4 11:53 S20bootlogs -> ../init.d/bootlogs
lrwxrwxrwx 1 root root 18 Jun 4 11:53 S21rc.local -> ../init.d/rc.local
lrwxrwxrwx 1 root root 19 Jun 4 11:53 S21rmnologin -> ../init.d/rmnologin
lrwxrwxrwx 1 root root 23 Jun 4 11:53 S21stop-bootlogd -> ../init.d/stop-bootlogd
student@vm1:/etc$ sudo rcconf --list
rsyslog on
ssh on
bootlogs on
portmap on
sudo on
nfs-common on
udev on
console-setup on
kbd on
keyboard-setup on
acpid on
cron on
atd on
procps on
module-init-tools on
exim4 off
student@vm1:/etc$ sudo update-rc.d exim4 enable
update-rc.d: using dependency based boot sequencing
student@vm1:/etc$ ls -al /etc/rc2.d
total 12
drwxr-xr-x 2 root root 4096 Jun 27 11:43 .
drwxr-xr-x 68 root root 4096 Jun 25 18:43 ..
-rw-r--r-- 1 root root 677 Mar 27 05:50 README
lrwxrwxrwx 1 root root 17 Jun 4 11:53 S14portmap -> ../init.d/portmap
lrwxrwxrwx 1 root root 20 Jun 4 11:53 S15nfs-common -> ../init.d/nfs-common
lrwxrwxrwx 1 root root 17 Jun 4 11:53 S17rsyslog -> ../init.d/rsyslog
lrwxrwxrwx 1 root root 14 Jun 15 19:02 S17sudo -> ../init.d/sudo
lrwxrwxrwx 1 root root 15 Jun 4 11:53 S18acpid -> ../init.d/acpid
lrwxrwxrwx 1 root root 13 Jun 4 11:53 S18atd -> ../init.d/atd
lrwxrwxrwx 1 root root 14 Jun 4 11:53 S18cron -> ../init.d/cron
lrwxrwxrwx 1 root root 15 Jun 27 11:43 S18exim4 -> ../init.d/exim4
lrwxrwxrwx 1 root root 13 Jun 4 11:53 S18ssh -> ../init.d/ssh
lrwxrwxrwx 1 root root 18 Jun 4 11:53 S20bootlogs -> ../init.d/bootlogs
lrwxrwxrwx 1 root root 18 Jun 4 11:53 S21rc.local -> ../init.d/rc.local
lrwxrwxrwx 1 root root 19 Jun 4 11:53 S21rmnologin -> ../init.d/rmnologin
46
lrwxrwxrwx 1 root root 23 Jun 4 11:53 S21stop-bootlogd -> ../init.d/stop-bootlogd
student@vm1:/etc$ sudo rcconf --list
rsyslog on
ssh on
bootlogs on
portmap on
sudo on
nfs-common on
udev on
console-setup on
kbd on
exim4 on
keyboard-setup on
acpid on
cron on
atd on
procps on
module-init-tools on
student@vm1:/etc$
Explanation
Installs rcconf package which allows you to easily manage runlevels.
Prints out directory containing startup scripts for runlevel 2. Mail server exim4 is
enabled for now.
Just prints out services for the same runlevel. Notice that there are several services
which are not shown because they are considered system services. rcconf –list –
expert will list them all and many more which reside on different runlevels.
Disables automatic startup of exim4, the mail server.
Prints out directory containing startup scripts for runlevel 2. exim4 startup script is
now renamed from S18exim4 to K01exim4 which means that exim4 is stopped
(killed) when entering this runlevel. If exim4 was not running to begin with, nothing
happens.
Prints out services for runlevel 2. Service exim4 is now off.
Enables automatic startup of exim4.
Prints out directory containing startup scripts for runlevel 2 once more, exim4 in
enabled again.
Prints out services for runlevel 2. Status of exim4 changed to on as expected.
Extra credit
Read about Debian boot process here: http://www.debian.org/doc/manuals/debianreference/ch03.en.html
Try this: aptitude install sysv-rc-conf, sysv-rc-conf –list. Read man sysv-rcconf.
47
Exercise 16. Processes: working with
proccesses, ps, kill
A program in its simplest is a file on you hard disk which contain instructions for your
central processor to execute. When you start it, it is copied to memory and control is
passed to it. Executed program is called a process. In multitasking operating
systems like Linux you may start many instances of a program, so may start many
processes from one program which all will be running (executing) at the same time.
This is an outline of what happens when you running ls:
YOU
Type in ls and its arguments into your terminal emulator and press <ENTER>
Control is now passed to Bash
Bash
Locates ls on your hard disk
Forks itself to Bash clone, that is clones itself to the new location in memory
Becomes parent process to Bash clone
Control is now passed to the Bash clone
Bash clone
Becomes child process to Bash
Preserves parent Bash process environment
Knows that it is it a clone and reacts accordingly
Overwrites itself with ls
Control is now passed to ls
ls
Prints out a directory listing for you or returns an error
Returns exit code
Control is now passed to Bash
Bash
Assigns ls exit code to ? variable
Waits for your input
YOU
Are able to type something once again
Some process are not interactive like ls, but just work quietly in the background, like
ssh. Processes have number of possible states and there is a number of operations
which you may perform on them by means of the signal mechanism.
Firstly let us talk about states. If you type ps ax –forest it will print out all processes
and you will get something like this (some hardware-related processes skipped):
student@vm1:/etc$ ps --forest ax
PID TTY
STAT TIME COMMAND
1?
Ss 0:16 init [2]
297 ?
S<s 0:00 udevd --daemon
392 ?
S< 0:00 \_ udevd --daemon
399 ?
S< 0:00 \_ udevd --daemon
691 ?
Ss 0:00 /sbin/portmap
703 ?
Ss 0:00 /sbin/rpc.statd
862 ?
Sl 0:00 /usr/sbin/rsyslogd -c4
886 ?
Ss 0:00 /usr/sbin/atd
971 ?
Ss 0:00 /usr/sbin/acpid
978 ?
Ss 0:01 /usr/sbin/cron
1177 ?
Ss 0:00 /usr/sbin/sshd
6671 ?
Ss 0:00 \_ sshd: student [priv]
6675 ?
S
0:00
\_ sshd: student@pts/0
6676 pts/0 Ss 0:00
\_ -bash
7932 pts/0 R+ 0:00
\_ ps --forest ax
1191 ?
Ss 0:00 /usr/sbin/exim4 -bd -q30m
48
1210 tty2
1211 tty3
1212 tty4
1213 tty5
1214 tty6
6216 tty1
Ss+
Ss+
Ss+
Ss+
Ss+
Ss+
0:00 /sbin/getty 38400 tty2
0:00 /sbin/getty 38400 tty3
0:00 /sbin/getty 38400 tty4
0:00 /sbin/getty 38400 tty5
0:00 /sbin/getty 38400 tty6
0:00 /sbin/getty 38400 tty1
Let us go through this list column py column:
PID — process ID. Each process has unique number associated with it, which
serves for uniquely identifying it. This means no two processes can ever have the
same PID.
TTY — teletype emulator associated with the process, which allows a process to
exchange information with you.
STAT — current process status. This will be descussed in detail below.
TIME — this is the amount of time in minutes and seconds this process is executed
on CPU.
COMMAND — this is program name with arguments. Notice how /usr/sbin/sshd is
parent of sshd: student which in turn is parent of sshd: student@pts/0 which in
turn is parent of bash which in turn is parent of ps –forest ax. You need to go
deeper!
Now let us discuss possible process states, which I took from PROCESS STATE
CODES of man ps:
STATE Description
Uninterruptible sleep (usually IO). Process is busy or hung, and does not
D
respond to signals, for example beacause hard disk had crashed and read
operation can not be completed.
R
Running or runnable (on run queue). Process is being executed right now.
Interruptible sleep (waiting for an event to complete). For example terminal
S
processes and Bash are often in this state, waiting for you to type in
something.
T
Stopped, either by a job control signal or because it is being traced.
W
paging (not valid since the 2.6.xx kernel, so do not bother about this).
X
dead (should never be seen).
Defunct (“zombie”) process, terminated but not reaped by its parent. This
Z
happens with incorrectly terminated processes, or processes whos pared
terminated erroneously.
<
high-priority (not nice to other users)
N
low-priority (nice to other users)
L
has pages locked into memory (for real-time and custom IO)
is a session leader. Related processes in Linux are treated as a unit, and
s
have shared Session ID (SID). If Process ID (PID) = Session ID (SID), this
process will be a Session Leader.
l
is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
is in the foreground process group. Such procceses are allowed to input and
+
output to the teletype emulator, tty.
Now let us talk about a way to communicate with all those process. You can do it by
sending them signals. Signal may be thought like a way to kick a process in the butt
so process will hear you and make what you want it to. This is abbreviated list of
possible signals which I took from SIGNALS section of man kill:
49
Signal Num Action Description
0
0
n/a
Exit code indicates if a signal may be sent
HUP 1
exit
Hangup on controlling terminal or parent process died
INT
2
exit
Interrupt from keyboard
QUIT 3
core Quit from keyboard
ILL
4
core Illegal instruction
TRAP 5
core Trace/breakpoint trap
ABRT 6
core Abort signal from abort(3)
FPE 8
core Floating point exception
KILL 9
exit
Non-catchable, non-ignorable kill
SEGV 11 core Invalid memory reference
PIPE 13 exit
Broken pipe: write to pipe with no readers
ALRM 14 exit
Timer signal from alarm(2)
TERM 15 exit
Terminate process
Again, do not be intimidated with not understanding, for now you just need to know
about HUP, TERM and KILL. There is even a song about KILL signal, named Kill
dash nine. Also, notice this abort(3) and alarm(2) entries? They mean that you may
read corresponding man pages by typing man 3 abort and man 2 alarm.
Now you will learn how to list running processes and send signals to them.
Do this
1: ps x
2: ps a
3: ps ax
4: ps axue --forest
5: dd if=/dev/zero of=~/test.img bs=1 count=$((1024*1024*1024)) &
6: kill -s USR1 $!
7: <ENTER>
8: kill -s USR1 $!
9: <ENTER>
10: kill -s TERM $!
11: <ENTER>
What you should see
student@vm1:/etc$ ps x
PID TTY STAT TIME COMMAND
6675 ?
S 0:00 sshd: student@pts/0
6676 pts/0 Ss 0:00 -bash
8193 pts/0 R+ 0:00 ps x
student@vm1:/etc$ ps a
PID TTY STAT TIME COMMAND
1210 tty2 Ss+ 0:00 /sbin/getty 38400 tty2
1211 tty3 Ss+ 0:00 /sbin/getty 38400 tty3
1212 tty4 Ss+ 0:00 /sbin/getty 38400 tty4
1213 tty5 Ss+ 0:00 /sbin/getty 38400 tty5
1214 tty6 Ss+ 0:00 /sbin/getty 38400 tty6
6216 tty1 Ss+ 0:00 /sbin/getty 38400 tty1
6676 pts/0 Ss 0:00 -bash
8194 pts/0 R+ 0:00 ps a
student@vm1:/etc$ ps ax
PID TTY STAT TIME COMMAND
50
1?
Ss 0:16 init [2]
--- skipped --- skipped --- skipped --691 ?
Ss 0:00 /sbin/portmap
703 ?
Ss 0:00 /sbin/rpc.statd
862 ?
Sl 0:00 /usr/sbin/rsyslogd -c4
886 ?
Ss 0:00 /usr/sbin/atd
971 ?
Ss 0:00 /usr/sbin/acpid
978 ?
Ss 0:01 /usr/sbin/cron
1177 ?
Ss 0:00 /usr/sbin/sshd
1191 ?
Ss 0:00 /usr/sbin/exim4 -bd -q30m
1210 tty2 Ss+ 0:00 /sbin/getty 38400 tty2
1211 tty3 Ss+ 0:00 /sbin/getty 38400 tty3
1212 tty4 Ss+ 0:00 /sbin/getty 38400 tty4
1213 tty5 Ss+ 0:00 /sbin/getty 38400 tty5
1214 tty6 Ss+ 0:00 /sbin/getty 38400 tty6
6216 tty1 Ss+ 0:00 /sbin/getty 38400 tty1
6671 ?
Ss 0:00 sshd: student [priv]
6675 ?
S 0:00 sshd: student@pts/0
6676 pts/0 Ss 0:00 -bash
8198 pts/0 R+ 0:00 ps ax
student@vm1:/etc$ ps axue --forest
USER
PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
--- skipped --- skipped --- skipped --root
1 0.0 0.0 8356 820 ?
Ss Jun06 0:16 init [2]
root
297 0.0 0.0 16976 1000 ?
S<s Jun06 0:00 udevd --daemon
root
392 0.0 0.0 16872 840 ?
S< Jun06 0:00 \_ udevd --daemon
root
399 0.0 0.0 16872 836 ?
S< Jun06 0:00 \_ udevd --daemon
daemon 691 0.0 0.0 8096 532 ?
Ss Jun06 0:00 /sbin/portmap
statd 703 0.0 0.0 14384 868 ?
Ss Jun06 0:00 /sbin/rpc.statd
root
862 0.0 0.1 54296 1664 ?
Sl Jun06 0:00 /usr/sbin/rsyslogd -c4
daemon 886 0.0 0.0 18716 436 ?
Ss Jun06 0:00 /usr/sbin/atd
root
971 0.0 0.0 3920 644 ?
Ss Jun06 0:00 /usr/sbin/acpid
root
978 0.0 0.0 22400 880 ?
Ss Jun06 0:01 /usr/sbin/cron
root 1177 0.0 0.1 49176 1136 ?
Ss Jun06 0:00 /usr/sbin/sshd
root 6671 0.0 0.3 70496 3284 ?
Ss Jun26 0:00 \_ sshd: student [priv]
student 6675 0.0 0.1 70496 1584 ?
S Jun26 0:00 \_ sshd: student@pts/0
student 6676 0.0 0.6 23644 6536 pts/0 Ss Jun26 0:00
\_ -bash LANG=en_US.UTF-8 USER=student
LOGNAME=student HOM
student 8199 0.0 0.1 16312 1088 pts/0 R+ 17:07 0:00
\_ ps axue --forest TERM=screen-bce
SHELL=/bin/bas
101
1191 0.0 0.1 44148 1076 ?
Ss Jun06 0:00 /usr/sbin/exim4 -bd -q30m
root 1210 0.0 0.0 5932 616 tty2 Ss+ Jun06 0:00 /sbin/getty 38400 tty2
root 1211 0.0 0.0 5932 612 tty3 Ss+ Jun06 0:00 /sbin/getty 38400 tty3
root 1212 0.0 0.0 5932 612 tty4 Ss+ Jun06 0:00 /sbin/getty 38400 tty4
root 1213 0.0 0.0 5932 612 tty5 Ss+ Jun06 0:00 /sbin/getty 38400 tty5
root 1214 0.0 0.0 5932 616 tty6 Ss+ Jun06 0:00 /sbin/getty 38400 tty6
root 6216 0.0 0.0 5932 612 tty1 Ss+ Jun14 0:00 /sbin/getty 38400 tty1
student@vm1:/etc$ dd if=/dev/zero of=~/test.img bs=1 count=$((1024*1024*1024)) &
[1] 8200
student@vm1:/etc$ kill -s USR1 $!
student@vm1:/etc$ 1455424+0 records in
1455424+0 records out
1455424 bytes (1.5 MB) copied, 1.76646 s, 824 kB/s
student@vm1:/etc$ kill -s USR1 $!
student@vm1:/etc$ 3263060+0 records in
3263060+0 records out
51
3263060 bytes (3.3 MB) copied, 3.94237 s, 828 kB/s
student@vm1:/etc$ kill -s TERM $!
student@vm1:/etc$
[1]+ Terminated
dd if=/dev/zero of=~/test.img bs=1 count=$((1024*1024*1024))
student@vm1:/etc$
Explanation
Prints out those processes owned (started by) you.
Prints out only processes associated with a terminal (tty) and those owned (started
by) you.
Prints out all processes currently running.
Prints out all running processes in tree form and include additional information such
as associated username and environment where available. Notice that this
information is available only for processes owned (started by) you. To view
environment information for all process type in sudo ps axue –forest instead.
Starts creating a zero-filled file (filled with null characters, do not bother for now) and
is sent to background by specifying & in the end.
Queries dd for status.
Because bash is only able to print out something in response for your input, you
need to press <ENTER> (issue empty command).
Queries dd for status again.
Again you need to press <ENTER> to see the output.
Sends dd termination signal, so dd quits.
To see that this indeed had happened you need to press <ENTER> once again.
Extra credit
Read man ps, man kill.
Read The Life Cycle of Processes and study this picture Process flow.
Print and fill in Signal table. You may use Documentation from kernel.org.
52
Exercise 17. Job schedulers: cron, at
Often a need arises to execute a program on schedule. For example let us imagine
that you need to make a copy of your work every midnight. To accomplish this in
Linux there is a special program called cron. It is a demon, which means that is
starts when computer is booted up and sits silently in the background, executing
other programs for you when time comes. Cron has several configuration files,
system-wide, and one per each user. By default users do not have crontabs,
because nothing is scheduled for them. This are cron configuration file locations:
/etc/crontab — system-wide cron configuration file.
/var/spool/cron/crontabs/ — directory for storing user configuration files.
Now let us talk about cron configuration file format. If you run cat /etc/crontab you
will see this:
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# m h dom mon dow user command
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#
Its syntax is simple enough. Let us take the line
47 6
* * 7 root
test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly
and take it apart:
17
# On 17-minute of every hour
*
# Every day of month
*
# Every month of the year
*
# Every weekday
root
# As user root
cd /
# Execute command 'cd /'
&&
# If 'cd /' completed successfully, then
run-parts --report /etc/cron.hourly # Execute command 'run-parts --report /etc/cron.hourly'
Now I will sum up cron format:
* * * * * user
command to be executed
T T T T T (for system
| | | | |
crontab only)
| | | | |
| | | | '----- day of week (0 - 6) (0 is Sunday, or use names)
| | | '---------- month of year (1 - 12)
| | '--------------- day of month (1 - 31)
| '-------------------- hour (0 - 23)
'------------------------- min (0 - 59)
This is abbreviated list of possible characters for time specification:
Asterisk (*) — all values in the field, e.g. * in minutes field means every minute.
Slash (/) — defines increments of ranges. For example, 30-40/3 means run program
on 30-th minute and every 3 minutes thereafter until 40-th minute.
Percent (%) — in command field, all data after percent will be sent to the command
as standard input. Do not bother with this for now.
53
Comma (,) — specify a list, e.g. 30,40 in minutes field means 30th and 40th minute.
Hyphen (-) — a range. For example, 30-40 in minutes field means every minute
between 30th and 40th minute.
L — specify last something, for example it allows you specify last day of month.
Now I will give you more some examples:
#m
h dom mon dow user
command
# Every minute, every hour, every day of month, every month of year, every day of week
*
* * * * root
/bin/false
# Every minute in interval of 30-40 minutes, every hour, every day of month, every month of year,
every day of week
30-40 * * * * root
/bin/false
# Every 5 minutes in interval of 30-40 minutes, every hour, every day of month, every month of year,
every day of week
30-40/5 * * * * root
/bin/false
# Every 5 minutes in interval of 0-59 minutes, every hour, every day of month, every month of year,
every day of week
*/5
* * * * user
command to be executed
# Every minute, every hour, last day of month, every month of year, every day of week
*
* L * * root
/bin/false
# Every minute, every hour, every day of month, every month of year, Monday and Thursday
*
* * * 0,3 root
/bin/false
Ok, but how to install a crontab? This is list of the cron commands:
crontab -l — print out current crontab.
crontab -e — edit crontab for current user.
crontab -r — remove crontab for current user.
crontab /path/to/file — install crontab for the current user, overwriting existing one
in the process.
crontab > /path/to/file — save crontab to a file.
Well, that was about using cron system daemon. But there is one more option to
schedule program execution. It is via at utility. The difference between them is that
cron is designed to run tasks repetitively, many times, and at is designed for
scheduling one-time tasks. This are relevant commands:
at — executes commands at a specified time.
atq — lists pending jobs.
atrm — removes a job.
batch — executes commands then system in idling.
As if this infodump is not enough, now I will give you a table of possible time
specification for at, taken from http://content.hccfl.edu/pollock/unix/atdemo.htm In the
examples below assume that the date and time currently is 10:00 AM Tuesday,
September 18, 2001
Example
Meaning
at noon
12:00 PM September 18, 2001
at midnight
12:00 AM September 19, 2001
at teatime
4:00 PM September 18, 2001
at tomorrow
10:00 AM September 19, 2001
at noon tomorrow
12:00 PM September 19, 2001
at next week
10:00 AM September 25, 2001
at next monday
10:00 AM September 24, 2001
at fri
10:00 AM September 21, 2001
at OCT
10:00 AM October 18, 2001
54
at 9:00 AM
9:00 AM September 19, 2001
at 2:30 PM
2:30 PM September 18, 2001
at 1430
2:30 PM September 18, 2001
at 2:30 PM tomorrow 2:30 PM September 19, 2001
at 2:30 PM next month 2:30 PM October 18, 2001
at 2:30 PM Fri
2:30 PM September 21, 2001
at 2:30 PM 9/21
2:30 PM September 21, 2001
at 2:30 PM Sept 21
2:30 PM September 21, 2001
at 2:30 PM 9/21/2010 2:30 PM September 21, 2010
at 2:30 PM 9.21.10
2:30 PM September 21, 2010
at now + 30 minutes 10:30 AM September 18, 2001
at now + 1 hour
11:00 AM September 18, 2001
at now + 2 days
10:00 AM September 20, 2001
at 4 PM + 2 days
4:00 PM September 20, 2001
at now + 3 weeks
10:00 AM October 9, 2001
at now + 4 months
10:00 AM January 18, 2002
at now + 5 years
10:00 AM September 18, 2007
Now you will learn how to add, view and remove at and crontab jobs.
Do this
1: echo 'echo Here I am, sitting in ur at, staring at ur date: $(date) | write student' | at now + 1 minutes
2: atq
Wait for your message to appear, press <ENTER> and type some more:
3: echo '* * * * * echo Here I am, sitting in ur crontab, staring at ur date: $(date) | write student' >
~/crontab.tmp
4: crontab -l
5: crontab ~/crontab.tmp
6: crontab -l
Now wait for this one message to appear and remove it:
7: crontab -r
8: crontab -l
What you should see
student@vm1:~$ echo 'echo Here I am, sitting in ur at, staring at ur date: $(date) | write student' | at now + 1
minutes
warning: commands will be executed using /bin/sh
job 13 at Thu Jun 28 14:43:00 2012
student@vm1:~$ atq
14 Thu Jun 28 14:45:00 2012 a student
student@vm1:~$
Message from student@vm1 on (none) at 14:43 ...
Here I am, sitting in ur at, staring at ur date: Thu Jun 28 14:43:00 MSK 2012
EOF
student@vm1:~$ crontab -l
no crontab for student
student@vm1:~$ echo '* * * * * echo Here I am, sitting in ur crontab, staring at ur date: $(date) | write
student' > ~/crontab.tmp
student@vm1:~$ crontab -l
* * * * * echo Here I am, sitting in ur crontab, staring at ur date: $(date) | write student
55
student@vm1:~$
Message from student@vm1 on (none) at 14:47 ...
Here I am, sitting in ur crontab, staring at ur date: Thu Jun 28 14:47:01 MSK 2012
EOF
student@vm1:~$ crontab -r
student@vm1:~$ crontab -l
no crontab for student
student@vm1:~$
Explanation
Makes at execute the command echo Here I am, sitting in ur at, staring at ur
date: $(date) | write student when next minute starts.
Prints out at job queue.
Writes line echo '* * * * * echo Here I am, sitting in ur crontab, staring at ur date:
$(date) | write student to the file crontab.tmp in your home direcroty.
Prints out your current crontab, but there is none currently so it just tells this to you.
Loads contents of crontab.tmp to your personal crontab file.
Prints out your current crontab. Now there is something in it.
Removes your current crontab.
Tells you that you do not have a crontab once again.
Extra credit
Read man crontab, man at, man write.
Make your system tell you current time every 5 minutes.
Make your system tell you current hour at the start of every hour.
56
Exercise 18. Logging, /var/log, rsyslog,
logger
Daemon are those programs which run in the background. Because of this a
question arises: how do they tell you how they are doing? And how do they tell you
that something got wrong? This problem is solved by log files, in which daemons
write their status and operations. In Debian this files reside in /var/log directory.
But who writes those files? The most obvious answer, The daemons themselves, is
actually often wrong. In some cases daemons do indeed write log files themselves,
but usually they do it through the daemon called rsyslogd, which is called a logging
daemon. It writes logs to different files for simplifying searching and analyzing them.
Do distinguish between this files it has a concept called facilities. This is list of
standard facilities:
Facility Facility Description
Facility Facility Description
auth
Authorization-related messages
local0 local use 0
authpriv Security information of sensitive nature
local1 local use 1
cron
Cron information
local2 local use 2
daemon System daemons
local3 local use 3
ftp
FTP demon messagestion messages
local4 local use 4
kern
Kernel messages
local5 local use 5
lpr
Line printer subsystem
local6 local use 6
mail
Mail subsystem
local7 local use 7
news
News subsystem
security Depreacated name for auth
syslog Messages generated internally by syslogd
user
uucp
UUCP subsystem
Every entry is also marked with severity status for ease of analyzing what happaned:
Code
Severity
Description
General Description
Name
Should be corrected immediately, therefore
Action must be
notify staff who can fix the problem. An
alert
Alert
taken
example would be the loss of a backup ISP
immediately.
connection.
Should be corrected immediately, but
Critical
crit
Critical
indicates failure in a primary system, an
conditions.
example is a loss of primary ISP connection.
Debug-level
Info useful to developers for debugging the
debug Debug
messages.
application, not useful during operations.
A “panic” condition usually affecting multiple
System is
emerg Emergency
apps/servers/sites. At this level it would
unusable.
usually notify all tech staff on call.
Non-urgent failures, these should be relayed
err
Error
Error conditions.
to developers or admins; each item must be
57
resolved within a given time.
Non-urgent failures, these should be relayed
Deprecated
error
Error
to developers or admins; each item must be
name for err
resolved within a given time.
Normal operational messages - may be
Informational
info
Informational
harvested for reporting, measuring
messages.
throughput, etc. - no action required.
Events that are unusual but not error
Normal but
conditions - might be summarized in an
notice Notice
significant
email to developers or admins to spot
condition.
potential problems - no immediate action
required.
A “panic” condition usually affecting multiple
Deprecated
panic
Emergency
apps/servers/sites. At this level it would
name for emerg
usually notify all tech staff on call.
Warning messages, not an error, but
Warning
indication that an error will occur if action is
warning Warning
conditions.
not taken, e.g. file system 85% full - each
item must be resolved within a given time.
Warning messages, not an error, but
Deprecated
indication that an error will occur if action is
warn
Warning
name for
not taken, e.g. file system 85% full - each
warning
item must be resolved within a given time.
Because if log files are left to themselves they tend to grow really big and consume
all available disk space, there is a mechanism called rotation. This mechanism by
default usually keeps only seven last log files, for last 7 days, including today.
Rotation is performed by logrotate daemon, to help you understand what this
daemon does I wrote it out for you:
Day 0
log.0 is created
Day 1
mv log.0 log.1
log.0 is created
Day 2
mv log.1 log.2
mv log.0 log.1
log.0 is created
Day 3
mv log.2 log.3
mv log.1 log.2
mv log.0 log.1
log.0 is created
Day 4
mv log.3 log.4
mv log.2 log.3
mv log.1 log.2
mv log.0 log.1
log.0 is created
Day 5
mv log.4 log.5
mv log.3 log.4
mv log.2 log.3
mv log.1 log.2
58
mv log.0 log.1
log.0 is created
Day 6
mv log.5 log.6
mv log.4 log.5
mv log.3 log.4
mv log.2 log.3
mv log.1 log.2
mv log.0 log.1
log.0 is created
Day 7
rm log.6
mv log.5 log.6
mv log.4 log.5
mv log.3 log.4
mv log.2 log.3
mv log.1 log.2
mv log.0 log.1
log.0 is created
Let me reiterate for you:
Logging is the process of recording events, with an automated computer program, in
a certain scope in order to provide an audit trail that can be used to understand the
activity of the system and to diagnose problems.
Logging daemon is a program which other programs may ask to write something
down in a log file.
Every log entry has facility (logging category) and severity (how important it is)
attributes.
Rotation is a process of keeping only limited number of log files to avoid disks filling
up.
In Debian, log files are usually found in /var/log directory.
This are useful commands to work with logs (remember to open relevant man pages
and find out what those program options are for):
logger Hello, I have a kitty! — write a custom log message.
ls -altr /var/log — list log directory in such a way that lastly modified files are in the
end.
grep student /var/log/auth.log — list all lines in a file which contain student.
grep -irl student /var/log — list all files which contain student anywhere.
find /var/log -mmin -10 — find any files which are modified in last 10 minutes.
tail /var/log/auth.log — print out last 10 lines of a log file.
tail -f /var/log/auth.log — follow a log file in real time. It is very useful when
configuring daemons.
No you will learn how to view logs and write something to system log.
Do this
1: sudo -s
2: cd /var/log
3: ls -altr | tail
4: tail auth.log
5: grep student auth.log | tail
6: /etc/init.d/exim4 restart
7: find /var/log -mmin -5
8: tail /var/log/exim4/mainlog
9: grep -irl rcconf .
10: tail ./dpkg.log
11: last
12: lastlog
59
13: logger local0.alert I am a kitty, sittin in ur system watchin u work ^^
14: ls -altr | tail
15: tail messages
What you should see
student@vm1:~$ sudo -s
root@vm1:/home/student# cd /var/log
root@vm1:/var/log# ls -altr | tail
-rw-r----- 1 root
adm 46955 Jun 29 12:28 messages
-rw-r----- 1 root
adm 19744 Jun 29 12:28 dmesg
-rw-r----- 1 root
adm 696 Jun 29 12:28 daemon.log
drwxr-xr-x 7 root
root 4096 Jun 29 12:28 .
-rw-r----- 1 root
adm 60738 Jun 29 12:28 syslog
-rw-r----- 1 root
adm 58158 Jun 29 12:28 kern.log
-rw-r----- 1 root
adm 12652 Jun 29 12:28 debug
-rw-rw-r-- 1 root
utmp 75264 Jun 29 12:28 wtmp
-rw-rw-r-- 1 root
utmp 292584 Jun 29 12:28 lastlog
-rw-r----- 1 root
adm 38790 Jun 29 12:40 auth.log
root@vm1:/var/log# tail auth.log
Jun 29 12:28:22 vm1 sshd[983]: Server listening on 0.0.0.0 port 22.
Jun 29 12:28:22 vm1 sshd[983]: Server listening on :: port 22.
Jun 29 12:28:44 vm1 sshd[1214]: Accepted password for student from 194.85.195.183 port 53775 ssh2
Jun 29 12:28:44 vm1 sshd[1214]: pam_unix(sshd:session): session opened for user student by (uid=0)
Jun 29 12:30:49 vm1 sudo: student : TTY=pts/0 ; PWD=/home/student ; USER=root ; COMMAND=/bin/bash
Jun 29 12:30:53 vm1 login[1260]: pam_securetty(login:auth): unexpected response from failed conversation
function
Jun 29 12:30:53 vm1 login[1260]: pam_securetty(login:auth): cannot determine username
Jun 29 12:35:08 vm1 sudo: student : TTY=pts/0 ; PWD=/home/student ; USER=root ; COMMAND=/bin/bash
Jun 29 12:35:14 vm1 sudo: student : TTY=pts/0 ; PWD=/home/student ; USER=root ; COMMAND=/bin/bash
Jun 29 12:40:32 vm1 sudo: student : TTY=pts/0 ; PWD=/home/student ; USER=root ; COMMAND=/bin/bash
root@vm1:/var/log# tail auth.log | grep student
Jun 29 12:28:44 vm1 sshd[1214]: Accepted password for student from 194.85.195.183 port 53775 ssh2
Jun 29 12:28:44 vm1 sshd[1214]: pam_unix(sshd:session): session opened for user student by (uid=0)
Jun 29 12:30:49 vm1 sudo: student : TTY=pts/0 ; PWD=/home/student ; USER=root ; COMMAND=/bin/bash
Jun 29 12:35:08 vm1 sudo: student : TTY=pts/0 ; PWD=/home/student ; USER=root ; COMMAND=/bin/bash
Jun 29 12:35:14 vm1 sudo: student : TTY=pts/0 ; PWD=/home/student ; USER=root ; COMMAND=/bin/bash
Jun 29 12:40:32 vm1 sudo: student : TTY=pts/0 ; PWD=/home/student ; USER=root ; COMMAND=/bin/bash
root@vm1:/var/log# grep student auth.log | tail
Jun 29 12:26:33 vm1 sshd[1302]: Accepted password for student from 194.85.195.183 port 53008 ssh2
Jun 29 12:26:33 vm1 sshd[1302]: pam_unix(sshd:session): session opened for user student by (uid=0)
Jun 29 12:26:38 vm1 sudo: student : TTY=pts/0 ; PWD=/home/student ; USER=root ; COMMAND=/bin/bash
Jun 29 12:28:02 vm1 sshd[1302]: pam_unix(sshd:session): session closed for user student
Jun 29 12:28:44 vm1 sshd[1214]: Accepted password for student from 194.85.195.183 port 53775 ssh2
Jun 29 12:28:44 vm1 sshd[1214]: pam_unix(sshd:session): session opened for user student by (uid=0)
Jun 29 12:30:49 vm1 sudo: student : TTY=pts/0 ; PWD=/home/student ; USER=root ; COMMAND=/bin/bash
Jun 29 12:35:08 vm1 sudo: student : TTY=pts/0 ; PWD=/home/student ; USER=root ; COMMAND=/bin/bash
Jun 29 12:35:14 vm1 sudo: student : TTY=pts/0 ; PWD=/home/student ; USER=root ; COMMAND=/bin/bash
Jun 29 12:40:32 vm1 sudo: student : TTY=pts/0 ; PWD=/home/student ; USER=root ; COMMAND=/bin/bash
root@vm1:/home/student# /etc/init.d/exim4 restart
Stopping MTA for restart: exim4_listener.
Restarting MTA: exim4.
root@vm1:/home/student# find /var/log -mmin -5
/var/log/exim4/mainlog
/var/log/auth.log
root@vm1:/home/student# tail /var/log/exim4/mainlog
2012-06-29 12:24:11 exim 4.72 daemon started: pid=1159, -q30m, listening for SMTP on [127.0.0.1]:25 [::1]:25
2012-06-29 12:24:11 Start queue run: pid=1165
60
2012-06-29 12:24:11 End queue run: pid=1165
2012-06-29 12:28:22 exim 4.72 daemon started: pid=1190, -q30m, listening for SMTP on [127.0.0.1]:25 [::1]:25
2012-06-29 12:28:22 Start queue run: pid=1196
2012-06-29 12:28:22 End queue run: pid=1196
2012-06-29 12:41:18 exim 4.72 daemon started: pid=1622, -q30m, listening for SMTP on [127.0.0.1]:25 [::1]:25
2012-06-29 12:41:18 Start queue run: pid=1624
2012-06-29 12:41:18 End queue run: pid=1624
2012-06-29 12:42:28 exim 4.72 daemon started: pid=1886, -q30m, listening for SMTP on [127.0.0.1]:25 [::1]:25
root@vm1:/home/student# grep -irl rcconf .
./aptitude
./apt/history.log
./apt/term.log
./dpkg.log
./auth.log
root@vm1:/home/student# tail ./dpkg.log
2012-06-26 19:27:40 status unpacked rcconf 2.5
2012-06-26 19:27:40 status unpacked rcconf 2.5
2012-06-26 19:27:40 trigproc man-db 2.5.7-8 2.5.7-8
2012-06-26 19:27:40 status half-configured man-db 2.5.7-8
2012-06-26 19:27:40 status installed man-db 2.5.7-8
2012-06-26 19:27:41 startup packages configure
2012-06-26 19:27:41 configure rcconf 2.5 2.5
2012-06-26 19:27:41 status unpacked rcconf 2.5
2012-06-26 19:27:41 status half-configured rcconf 2.5
2012-06-26 19:27:41 status installed rcconf 2.5
root@vm1:/var/log# last
student pts/0
sis.site Fri Jun 29 12:26 still logged in
student pts/0
sis.site Fri Jun 29 12:14 - down (00:09)
student pts/0
sis.site Thu Jun 28 19:40 - 11:25 (15:45)
student pts/0
sis.site Wed Jun 27 19:14 - 17:04 (21:50)
student pts/0
sis.site Tue Jun 26 13:54 - 18:18 (1+04:23)
student pts/0
sis.site Thu Jun 21 15:23 - 13:11 (4+21:47)
student pts/0
sis.site Fri Jun 15 19:34 - 12:01 (5+16:26)
student pts/0
sis.site Fri Jun 15 19:11 - 19:34 (00:22)
reboot system boot 2.6.32-5-amd64 Fri Jun 29 12:24 - 12:26 (00:02)
student pts/0
sis.site Fri Jun 29 12:14 - down (00:09)
root@vm1:/var/log# lastlog
Username
Port From
Latest
root
**Never logged in**
daemon
**Never logged in**
bin
**Never logged in**
sys
**Never logged in**
sync
**Never logged in**
games
**Never logged in**
man
**Never logged in**
lp
**Never logged in**
mail
**Never logged in**
news
**Never logged in**
uucp
**Never logged in**
proxy
**Never logged in**
www-data
**Never logged in**
backup
**Never logged in**
list
**Never logged in**
irc
**Never logged in**
gnats
**Never logged in**
nobody
**Never logged in**
libuuid
**Never logged in**
61
Debian-exim
**Never logged in**
statd
**Never logged in**
sshd
**Never logged in**
student
pts/0 sis.site Fri Jun 29 12:28:45 +0400 2012
root@vm1:/var/log# logger local0.alert I am a kitty, sittin in ur system watchin u work ^^
root@vm1:/var/log# ls -altr | tail
-rw-r----- 1 root
adm 696 Jun 29 12:28 daemon.log
drwxr-xr-x 7 root
root 4096 Jun 29 12:28 .
-rw-r----- 1 root
adm 58158 Jun 29 12:28 kern.log
-rw-r----- 1 root
adm 12652 Jun 29 12:28 debug
-rw-rw-r-- 1 root
utmp 75264 Jun 29 12:28 wtmp
-rw-rw-r-- 1 root
utmp 292584 Jun 29 12:28 lastlog
-rw-r----- 1 root
adm 38971 Jun 29 13:17 auth.log
-rw-r----- 1 root
adm 229 Jun 29 13:19 user.log
-rw-r----- 1 root
adm 60932 Jun 29 13:19 syslog
-rw-r----- 1 root
adm 47047 Jun 29 13:19 messages
root@vm1:/var/log# tail messages
Jun 29 12:28:21 vm1 kernel: [ 1.846975] processor LNXCPU:00: registered as cooling_device0
Jun 29 12:28:21 vm1 kernel: [ 1.868828] usbcore: registered new interface driver hiddev
Jun 29 12:28:21 vm1 kernel: [ 1.895676] input: QEMU 0.14.1 QEMU USB Tablet as
/devices/pci0000:00/0000:00:01.2/usb1/1-1/1-1:1.0/input/input4
Jun 29 12:28:21 vm1 kernel: [ 1.895743] generic-usb 0003:0627:0001.0001: input,hidraw0: USB HID v0.01
Pointer [QEMU 0.14.1 QEMU USB Tablet] on usb-0000:00:01.2-1/input0
Jun 29 12:28:21 vm1 kernel: [ 1.895762] usbcore: registered new interface driver usbhid
Jun 29 12:28:21 vm1 kernel: [ 1.895765] usbhid: v2.6:USB HID core driver
Jun 29 12:28:21 vm1 kernel: [ 2.373061] EXT3 FS on vda1, internal journal
Jun 29 12:28:21 vm1 kernel: [ 2.394992] loop: module loaded
Jun 29 12:28:21 vm1 kernel: [ 2.413478] input: ImExPS/2 Generic Explorer Mouse as
/devices/platform/i8042/serio1/input/input5
Jun 29 13:19:11 vm1 student: local0.alert I am a kitty, sittin in ur system watchin u work ^^
root@vm1:/var/log#
Explanation
Opens root (superuser) shell. This is because you are not allowed to read all log files
due security considerations when working as student.
Changes directory to /var/log.
Prints out all files sorted by date, with last modified ones at the bottom.
Prints out last 10 lines from auth.log, which contains information about logging into
the system.
Prints out last 10 lines from auth.log which contain student.
Restars exim4 mail server.
Prints out files changes in last 5 minutes. Now you can easily spot in which file
exim4 logs its operation.
Prints out last 10 lines from exim4 log.
Searches for rcconf through all files in current directory. Now you can easily spot
where Debian package system logs its operations.
Pritns out last 10 lines from dpkg.log, which contain information about package
installation and removal.
Prints out information about last logins of users.
Prints out information about the most recent logins of all users.
Passes your message to rsyslogd daemon.
Prints out all files sorted by date, with last modified ones at the bottom, again. Now
you may see there did your message go.
62
Prints out last 10 lines from messages, you are able to see that your message is
indeed logged.
Extra credit
Read man pages for rsyslogd and logger.
Find out the difference between last and lastlog by reading corresponding man
pages.
Read logrotate man page and remember that it exists.
Execute tail -f /var/log/auth.log and make second connection to vm1 (with putty if
you are working on Windows). Nice, huh?
63
Exercise 19. Filesystems: mounting,
mount, /etc/fstab
I hope you are familiar with concept of partitions. In case if you are not I will
introduce you briefly to it. First, quote from Wikipedia:
Disk partitioning is the act of dividing a hard disk drive into multiple logical storage
units referred to as partitions, to treat one physical disk drive as if it were multiple
disks.
Take a look:
student@vm1:~$ sudo parted /dev/vda
GNU Parted 2.3
Using /dev/vda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit GB
(parted) p
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 17.2GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type
File system Flags
1
0.00GB 13.3GB 13.3GB extended
5
0.00GB 1.02GB 1.02GB logical ext3
boot
6
1.03GB 2.05GB 1.02GB logical linux-swap(v1)
7
2.05GB 3.07GB 1.02GB logical ext3
8
3.07GB 5.12GB 2.05GB logical ext3
9
5.12GB 9.22GB 4.09GB logical ext3
10
9.22GB 13.3GB 4.09GB logical ext3
(parted)
This is one physical hard disk which is divided in 7 different partitions. The reasons
to do do are numerous, but best understood as an application of “divide and
conquer” principle. When divided in such a way one rogue program can not bring
whole sever down by consuming all disk space, this program will be limited to its
partition. I will not talk about partitioning disks just just yet, but I will continue taking
about file systems, by quoting Wikipedia again:
A file system (or filesystem) is a means to organize data expected to be retained
after a program terminates by providing procedures to store, retrieve and update
data, as well as manage the available space on the device(s) which contain it. A file
system organizes data in an efficient manner and is tuned to the specific
characteristics of the device. A tight coupling usually exists between the operating
system and the file system. Some file systems provide mechanisms to control
access to the data and metadata. Ensuring reliability is a major responsibility of a file
system. Some file systems allow multiple programs to update the same file at nearly
the same time.
Unix-like operating systems create a virtual file system, which makes all the files on
all the devices appear to exist in a single hierarchy. This means, in those systems,
there is one root directory, and every file existing on the system is located under it
somewhere. Unix-like systems can use a RAM disk or network shared resource as
its root directory.
What this means is that all file systems are integrated in one big tree. For those
familiar with Microsoft Windows this means that instead C:, D:, and so on naming
64
scheme there is a single root, /, to which all other partitions connect. The process of
connecting a file system to existing directory is called mounting. A directory to which
a file system is connected is called a mount point. Again, take a look:
student@vm1:~$ mount
/dev/vda5 on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/vda10 on /home type ext3 (rw)
/dev/vda7 on /tmp type ext3 (rw)
/dev/vda9 on /usr type ext3 (rw)
/dev/vda8 on /var type ext3 (rw)
This are the same partitions I showed you earlier in this list you their mount points.
Ones which do not start with /dev/vda are virtual file systems, which allow to access
different system facilities, but they are not relevant for this exercise. Now let us take
a look at /etc/fstab file:
student@vm1:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options>
<dump> <pass>
proc
/proc
proc defaults 0
0
# / was on /dev/vda5 during installation
UUID=128559db-a2e0-4983-91ad-d4f43f27da49 /
ext3 errors=remount-ro 0
1
# /home was on /dev/vda10 during installation
UUID=32852d29-ddee-4a8d-9b1e-f46569a6b897 /home ext3 defaults
0
2
# /tmp was on /dev/vda7 during installation
UUID=869db6b4-aea0-4a25-8bd2-f0b53dd7a88e /tmp
ext3 defaults
0
2
# /usr was on /dev/vda9 during installation
UUID=0221be16-496b-4277-b131-2371ce097b44 /usr
ext3 defaults
0
2
# /var was on /dev/vda8 during installation
UUID=2db00f94-3605-4229-8813-0ee23ad8634e /var
ext3 defaults
0
2
# swap was on /dev/vda6 during installation
UUID=3a936af2-2c04-466d-b98d-09eacc5d104c none
swap sw
0
0
/dev/scd0
/media/cdrom0 udf,iso9660 user,noauto 0
0
Looks scary, but let us take one line:
# <file system>
<mount point> <type> <options>
<dump> <pass>
UUID=128559db-a2e0-4983-91ad-d4f43f27da49 /
ext3 errors=remount-ro 0
1
And take it apart field by field:
UUID=128559db-a2e0-4983-91ad-d4f43f27da49 # Filesystem to mount. This UUID is synonim for
/dev/vda5
/
# This is root filesystem, mount it to /
ext3
# This is ext3 filesystem. There are many different filesystems out there
errors=remount-ro # If any errors encountered during mounting filesystem should be remounted
read-only
0
# This filesystem should not be backed up by dump utility
1
# This filesystem should be checked first by fsck utility
As always, this information is available to you via man fstab. Now I will show you
several commands for working with existing file systems:
mount — print out all mounted filesystems.
mount -a — mount all filesystem described in /etc/fstab.
65
mount /dev/sda<N> /<mount point> — mount a partition.
umount /dev/sda<N> /<mount point> — mount a partition.
mount -h — print out short help about using mount.
fsck — check partition for errors.
blkid — print out unique partition identifiers.
Now you will learn how to list mounted partitions, mount and unmount them.
Do this
1: cat /etc/fstab
2: mount
3: sudo blkid
4: sudo umount /tmp
5: mount
6: sudo fsck /tmp
7: sudo mount -a
8: mount
What you should see
student@vm1:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options>
<dump> <pass>
proc
/proc
proc defaults
0
0
# / was on /dev/sda1 during installation
UUID=05d469bb-dbfe-4d5a-9bb2-9c0fe9fa8577 /
ext3 errors=remount-ro 0
# /home was on /dev/sda9 during installation
UUID=a1b936a0-df38-4bf5-b095-6220ffdfc63c /home
ext3 defaults
0
2
# /tmp was on /dev/sda8 during installation
UUID=d0a86453-0dbb-4f33-a023-6c09fe9fa202 /tmp ext3 defaults 0 2
# /usr was on /dev/sda5 during installation
UUID=b9544cbb-cdb6-4f3b-89e7-a339f52bfac7 /usr
ext3 defaults
0
2
# /var was on /dev/sda6 during installation
UUID=e15e713b-5850-4bc3-b99e-ab6f1d037caa /var
ext3 defaults
0
2
# swap was on /dev/sda7 during installation
UUID=4d516f09-80ff-4956-8a75-e9757697f6b1 none
swap sw
0
0
/dev/scd0
/media/cdrom0 udf,iso9660 user,noauto 0
0
student@vm1:~$ mount
/dev/sda1 on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/sda9 on /home type ext3 (rw)
/dev/sda5 on /usr type ext3 (rw)
/dev/sda6 on /var type ext3 (rw)
/dev/sda8 on /tmp type ext3 (rw)
/dev/sda8 on /tmp type ext3 (rw)
student@vm1:~$ sudo blkid
/dev/sda1: UUID="05d469bb-dbfe-4d5a-9bb2-9c0fe9fa8577" TYPE="ext3"
/dev/sda5: UUID="b9544cbb-cdb6-4f3b-89e7-a339f52bfac7" TYPE="ext3"
66
1
/dev/sda6: UUID="e15e713b-5850-4bc3-b99e-ab6f1d037caa" TYPE="ext3"
/dev/sda7: UUID="4d516f09-80ff-4956-8a75-e9757697f6b1" TYPE="swap"
/dev/sda8: UUID="d0a86453-0dbb-4f33-a023-6c09fe9fa202" TYPE="ext3"
/dev/sda9: UUID="a1b936a0-df38-4bf5-b095-6220ffdfc63c" TYPE="ext3"
student@vm1:~$ sudo umount /tmp
student@vm1:~$ mount
/dev/sda1 on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/sda9 on /home type ext3 (rw)
/dev/sda5 on /usr type ext3 (rw)
/dev/sda6 on /var type ext3 (rw)
student@vm1:~$ sudo fsck /tmp
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/dev/sda8: clean, 11/61752 files, 13973/246784 blocks
student@vm1:~$ sudo mount -a
student@vm1:~$ mount
/dev/sda1 on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/sda9 on /home type ext3 (rw)
/dev/sda5 on /usr type ext3 (rw)
/dev/sda6 on /var type ext3 (rw)
/dev/sda8 on /tmp type ext3 (rw)
student@vm1:~$
Explanation
Prints out contents of your /etc/fstab file which contains information about your
partitions and where to mount them.
Prints out currently mounted partitions.
Prints out UUIDs of all partitions in your system.
Unmounts /tmp partition so you will be able to check it.
Prints out currently mounted partitions once again. /tmp is now absent from this list.
Check /tmp partition for errors. fsck knows which partition to check by reading
corresponding /etc/fstab entry.
Mounts all partitions described in /etc/fstab.
Prints out currently mounted partitions yet another time. /tmp has returned to this list.
Extra credit
Read man fstab, man mount.
Read http://tldp.org/LDP/sag/html/filesystems.html
67
Exercise 20. Filesystems: modifying
and creating filesystems, tune2fs, mkfs
Let me introduce you to filesystem-related terminology:
Filesystem — a means to organize data expected to be retained after a program
terminates by providing procedures to store, retrieve and update data, as well as
manage the available space on the device(s) which contain it.
Inode — an Index Node, is a structure which store all the information about a file
system object (file, directory, etc.) except data content and file name.
Block — a minimum chunk of disk space which can be allocated. It usually defaults
to 4096 bytes, or 4 Kilobytes.
Journal — a structure which allows the filesystem track of what was written and
when. This allows to quickly understand what was not properly written in case of
power outage or similar problem.
Next, let me give you an outline explaining in broad strokes how a filesystem works.
To access file by name Linux kernel:
Finds file name in directory containing the file.
Gets file Inode number.
Finds Inode by its number in Inodes area.
Reads locations of blocks containing the data from this Inode.
Using this locations reads file from this blocks in data area.
Now, every file system has many options associated with it. These options may be
viewed and changed via tune2fs program. Here is an annotated output of tune2fs -l
/dev/sda8:
student@vm1:~$ sudo tune2fs -l /dev/sda8
tune2fs 1.41.12 (17-May-2010)
# Just a label. Can be anything or nothing.
Filesystem volume name: <none>
# Here should be last mount point.
Last mounted on:
<not available>
# Unique file system identifier.
Filesystem UUID:
869db6b4-aea0-4a25-8bd2-f0b53dd7a88e
# A special number at known place, which defines FS type. Try this:
# sudo dd if=/dev/sda8 of=/dev/stdout bs=1 skip=1080 count=2 | hexdump -C
Filesystem magic number: 0xEF53
# FS version
Filesystem revision #: 1 (dynamic)
# Enabled FS features
Filesystem features:
# It is a journaling file system. A journaling file system is a file system that
# keeps track of the changes that will be made in a journal (usually a circular log in
# a dedicated area of the file system) before committing them to the main file system.
# In the event of a system crash or power failure, such file systems are quicker to bring
# back online and less likely to become corrupted.
# http://en.wikipedia.org/wiki/Journaling_file_system
has_journal
# Has extended attributes like extended ACL.
ext_attr
# Reserves space for system information, which allows FS resizing.
resize_inode
# Use indexes to speed up look-ups in large directories.
dir_index
# Store file type information in directory entries.
68
filetype
# Means that fsck needs to be run.
needs_recovery
# Fewer superblock backup copies, saves space on large FS.
sparse_super
# Can contain files > 2GB. Kernel sets this up automatically on >2GB file creation.
large_file
# Which hash to use in dir_index.
Filesystem flags:
signed_directory_hash
# Which options to use when mounting.
Default mount options: (none)
# Need to run fsck?
Filesystem state:
clean
# What to do on error? Continue, remount read-only or panic?
Errors behavior:
Continue
# Which OS uses this FS
Filesystem OS type:
Linux
# Number of inodes total. Inode means "Index Node" and is a structure which
# store all the information about a file system object (file, directory, etc.) execpt
# data content and file name. The implication of this is that you can not have more files
# than you have Inodes.
# This is Inode structure which describes what information which is stored in Inode:
#/* Inode structure as it appears on an Inode table. */
#struct dinode
#{ ushort di_mode; /* mode and type of file */
# short di_nlink; /* number of links to file */
# ushort di_uid;
/* owner's user id
*/
# ushort di_gid;
/* owner's group id
*/
# off_t di_size; /* number of bytes in file */
# char di_addr[39]; /* disk block addresses */
# char di_gen;
/* file generation number */
# time_t di_atime; /* time last accessed
*/
# time_t di_mtime; /* time last modified
*/
# time_t di_ctime; /* time created
*/
#};
# This is also a good explanation:
# http://support.tux4u.nl/dokuwiki/lib/exe/fetch.php?media=linux:disk_bestandssysteem:inode.pdf
Inode count:
62464
# How many inodes are currently free.
Free inodes:
62452
# First block on device. Because each partition is represented as a separate device,
# this is set to 0.
First block:
0
# This is the Inode number of the first Inode in the file system.
First inode:
11
# Inode size in bytes. This is sometimes increased by default in newer Linux distributions,
# to permit storage of extended attributes in file, for example nanosecond timestamps.
Inode size:
256
# Requred space for additional Inode fields.
Required extra isize: 28
# Desired space for additional Inode fields. Does not matter because this size is
# requred anyway.
Desired extra isize:
28
# An invisible inode which stores file system journal.
Journal inode:
8
# Number of blocks total. A block is a minimum chunk of disk space which can be allocated.
# You can calculate partition size in GB with the following formula:
# Block count * Block size
# -----------------------#
1024^3
69
Block count:
249856
# How many blocks are reserved for superuser only. Ordinary users cannot use this
# reserved space. This is done for system remain operational in case some rogue
# program decides to fill up all available disk space.
Reserved block count: 12492
# How many blocks are currently free.
Free blocks:
241518
# An algorithm used for directory index, dir_index.
Default directory hash: half_md4
# As far as I can tell this is a seed value for dir_index hashing algorithm.
Directory Hash Seed:
d02c6099-bd06-4d29-a3d7-779df2aa2410
# Journal backup options.
Journal backup:
inode blocks
#
# Block size, in bytes. 4096 bytes means 4 KB here.
Block size:
4096
# Not implemented in ext3 FS. It is a feature which would allow to write several small files
# in one block to conserve space.
Fragment size:
4096
# Reserved space so Group Descriptor Table may grow in the future.
Reserved GDT blocks:
60
# Number of blocks per block group. Block groups contain a redundant copy of crucial filesystem
# control information (superblock and the FS descriptors) and also contains a
# part of the filesystem (a block bitmap, an Inode bitmap, a piece of the Inode table,
# and data blocks). The structure of a block group is represented in this table:
#,---------+---------+---------+---------+---------+---------,
#| Super | FS
| Block | Inode | Inode | Data |
#| block | desc. | bitmap | bitmap | table | blocks |
#`---------+---------+---------+---------+---------+---------'
# http://tldp.org/HOWTO/Filesystems-HOWTO-6.html
Blocks per group:
32768
# Number of fragments per group. Because there are no fragments in ext3 FS this is
# equal to Blocks per group.
Fragments per group:
32768
# Number of inodes per group.
Inodes per group:
7808
# The Inode blocks per group. Inode blocks an index into a teable that describe
# all file properties except the file name. It has references to the data blocks
# that contain the actual contents of the file.
# http://www.porcupine.org/forensics/forensic-discovery/chapter3.html
Inode blocks per group: 488
# FS Creation time.
Filesystem created:
Mon Jul 2 06:16:24 2012
# Last FS mount time.
Last mount time:
Mon Jul 2 06:57:21 2012
# Last FS write time.
Last write time:
Mon Jul 2 06:57:21 2012
# Number of times this FS was mounted.
Mount count:
6
# Number of counts before automatic checking. FS will be automatically checked if
# filesystem was mounted this number of times, or if check interval has come.
Maximum mount count:
34
# Last fsck run time
Last checked:
Mon Jul 2 06:16:24 2012
# Next FS check interval. FS will be checked if this interval has come,
# or if Maximum mount count is reached.
Check interval:
15552000 (6 months)
# Next FS check interval in human readable format.
Next check after:
Sat Dec 29 05:16:24 2012
# User ID for user which is able to user reserved space.
# It is root user (superuser) by default.
70
Reserved blocks uid:
0 (user root)
# Group ID for group which is able to user reserved space.
# It is root group by default.
Reserved blocks gid:
0 (group root)
Scary, huh? In practice you will find that only several parameters from this
description are actually useful, which are:
Reserved block count.
Maximum mount count.
Check interval.
Usually you would not want to modify other parameters, they are sane by default.
Here is a list of commands for working with file systems:
mkfs.ext3 — create an ext3 file system. If this command is executed on a device
with existing file system, this file system will be destroyed, so proceed with caution.
mkfs.ext4 — create an ext4 file system. This is actually the same program, try sudo
find /sbin -samefile sbin/mkfs.ext3.
tune2fs — print out and change file system parameters.
Now you will learn how to create new filesystem and modify its parameters.
Do this
1: sudo -s
2: umount /tmp
3: blkid | grep /dev/sda8
4: mkfs.ext3 /dev/sda8
5: blkid | grep /dev/sda8
6: blkid | grep /dev/sda8 >> /etc/fstab
7: vim /etc/fstab
Now you must replace UUID in line for /tmp:
# /tmp was on /dev/sda8 during installation
UUID=869db6b4-aea0-4a25-8bd2-f0b53dd7a88e /tmp
ext3
defaults
0
2
With one that you appended to the end of file:
/dev/sda8: UUID="53eed507-18e8-4f71-9003-bcea8c4fd2dd" TYPE="ext3" SEC_TYPE="ext2"
Because UUID are unique by definition yours will differ from mine. After replacing
this UUID, write file, quit and continue to type:
8: mount /tmp
9: tune2fs -c 2 /dev/sda8
10: unmount /tmp
11: fsck /tmp
12: for ((i=1;i<=4;i++)); do mount /tmp ; umount /tmp ; cat /var/log/messages | tail -n 4 ; done
13: fsck /tmp
14: mount -a
What you should see
student@vm1:~$ sudo -s
root@vm1:/home/student# umount /tmp
root@vm1:/home/student# blkid | grep /dev/sda8
/dev/sda8: UUID="869db6b4-aea0-4a25-8bd2-f0b53dd7a88e" TYPE="ext3" SEC_TYPE="ext2"
root@vm1:/home/student# mkfs.ext3 /dev/sda8
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
62464 inodes, 249856 blocks
12492 blocks (5.00%) reserved for the super user
First data block=0
71
Maximum filesystem blocks=255852544
8 block groups
32768 blocks per group, 32768 fragments per group
7808 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
root@vm1:/home/student# blkid | grep /dev/sda8
/dev/sda8: UUID="53eed507-18e8-4f71-9003-bcea8c4fd2dd" TYPE="ext3" SEC_TYPE="ext2"
root@vm1:/home/student# blkid | grep /dev/sda8 >> /etc/fstab
root@vm1:/home/student# vim /etc/fstab
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options>
<dump> <pass>
proc
/proc
proc defaults
0
0
# / was on /dev/vda5 during installation
UUID=128559db-a2e0-4983-91ad-d4f43f27da49 /
ext3 errors=re
# /home was on /dev/vda10 during installation
UUID=32852d29-ddee-4a8d-9b1e-f46569a6b897 /home
ext3 defaults
# /tmp was on /dev/sda8 during installation
UUID=869db6b4-aea0-4a25-8bd2-f0b53dd7a88e /tmp
ext3 defaults
# /usr was on /dev/vda9 during installation
UUID=0221be16-496b-4277-b131-2371ce097b44 /usr
ext3 defaults
# /var was on /dev/vda8 during installation
UUID=2db00f94-3605-4229-8813-0ee23ad8634e /var
ext3 defaults
# swap was on /dev/vda6 during installation
UUID=3a936af2-2c04-466d-b98d-09eacc5d104c none
swap sw
/dev/scd0
/media/cdrom0 udf,iso9660 user,noauto 0
0
/dev/sda8: UUID="53eed507-18e8-4f71-9003-bcea8c4fd2dd" TYPE="ext3" SEC_TYPE
22,1
Bot
#
# <file system> <mount point> <type> <options>
<dump> <pass>
proc
/proc
proc defaults
0
0
# / was on /dev/vda5 during installation
UUID=128559db-a2e0-4983-91ad-d4f43f27da49 /
ext3 errors=re
# /home was on /dev/vda10 during installation
UUID=32852d29-ddee-4a8d-9b1e-f46569a6b897 /home
ext3 defaults
# /tmp was on /dev/sda8 during installation
UUID=53eed507-18e8-4f71-9003-bcea8c4fd2dd /tmp
ext3 defaults
# /usr was on /dev/vda9 during installation
UUID=0221be16-496b-4277-b131-2371ce097b44 /usr
ext3 defaults
# /var was on /dev/vda8 during installation
UUID=2db00f94-3605-4229-8813-0ee23ad8634e /var
ext3 defaults
# swap was on /dev/vda6 during installation
UUID=3a936af2-2c04-466d-b98d-09eacc5d104c none
/dev/scd0
/media/cdrom0 udf,iso9660 user,noauto
"/etc/fstab" 22L, 1277C written
root@vm1:/home/student# mount /tmp
root@vm1:/home/student# tune2fs -c 2 /dev/sda8
72
0
swap sw
0
tune2fs 1.41.12 (17-May-2010)
Setting maximal mount count to 2
root@vm1:/home/student# unmount /tmp
root@vm1:/home/student# fsck /tmp
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/dev/sda8: clean, 11/62464 files, 8337/249856 blocks (check in 2 mounts)
root@vm1:/home/student# for ((i=1;i<=4;i++)); do mount /tmp ; umount /tmp ; cat /var/log/messages | tail -n
4 ; done
Jul 2 12:11:43 vm1 kernel: [21080.920658] EXT3-fs: mounted filesystem with ordered data mode.
Jul 2 12:11:58 vm1 kernel: [21096.363787] kjournald starting. Commit interval 5 seconds
Jul 2 12:11:58 vm1 kernel: [21096.364167] EXT3 FS on sda8, internal journal
Jul 2 12:11:58 vm1 kernel: [21096.364171] EXT3-fs: mounted filesystem with ordered data mode.
Jul 2 12:11:58 vm1 kernel: [21096.364171] EXT3-fs: mounted filesystem with ordered data mode.
Jul 2 12:11:58 vm1 kernel: [21096.381372] kjournald starting. Commit interval 5 seconds
Jul 2 12:11:58 vm1 kernel: [21096.381539] EXT3 FS on sda8, internal journal
Jul 2 12:11:58 vm1 kernel: [21096.381542] EXT3-fs: mounted filesystem with ordered data mode.
Jul 2 12:11:58 vm1 kernel: [21096.396152] kjournald starting. Commit interval 5 seconds
Jul 2 12:11:58 vm1 kernel: [21096.396158] EXT3-fs warning: maximal mount count reached, running e2fsck is
recommended
Jul 2 12:11:58 vm1 kernel: [21096.396344] EXT3 FS on sda8, internal journal
Jul 2 12:11:58 vm1 kernel: [21096.396348] EXT3-fs: mounted filesystem with ordered data mode.
Jul 2 12:11:58 vm1 kernel: [21096.412434] kjournald starting. Commit interval 5 seconds
Jul 2 12:11:58 vm1 kernel: [21096.412441] EXT3-fs warning: maximal mount count reached, running e2fsck is
recommended
Jul 2 12:11:58 vm1 kernel: [21096.412610] EXT3 FS on sda8, internal journal
Jul 2 12:11:58 vm1 kernel: [21096.412612] EXT3-fs: mounted filesystem with ordered data mode.
root@vm1:/home/student# fsck /tmp
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/dev/sda8 has been mounted 4 times without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda8: 11/62464 files (0.0% non-contiguous), 8337/249856 blocks
root@vm1:/home/student# mount -a
root@vm1:/home/student#
Explanation
Executes root (superuser) shell.
Unmounts /tmp, reading its location from /etc/fstab.
Prints out /dev/sda8 UUID, /dev/sda8 being the file system which mounts on /tmp.
Makes a new file system on /dev/sda8.
Prints out /dev/sda8 UUID one again, notice how it changed because you created a
new file system.
Appends this UUID to /etc/fstab.
Opens /etc/fstab for editing.
Mounts newly created file system. This is actually a check if you had replaced UUID
correctly, if not there will be an error message.
Sets checking /dev/sda8 every 2 mounts.
Unmounts /dev/sda8.
Checks /dev/sda8.
73
Mounts, unmounts /dev/sda8 and shows you last 4 lines from /var/log/messages
four times in a row. Notice how starting from 3-rd mount system notifies you that
e2fsck needs to be run. If you reboot your system, it would run e2fsck for you.
Checks /dev/sda8. fsck determines file system type and calls e2fsck for you
automatically.
Mounts all file systems. If there is no error, you successfully finished this exercise.
Extra credit
Read man mkfs, man mkfs.ext3, man tune2fs.
Read tune2fs -l listing at top of the page and read magic numbers for all your file
systems.
Calculate size of your file systems by hand using formula provided in block
description of tune2fs -l listing.
Read this presentation and do what is shown it in:
http://mcgrewsecurity.com/training/extx.pdf
74
Exercise 21. Filesystems: changing
root directory, chroot
Let me start with another Wikipedia quote:
A chroot on Unix operating systems is an operation that changes the apparent root
directory for the current running process and its children. A program that is run in
such a modified environment cannot name (and therefore normally not access) files
outside the designated directory tree. The term chroot may refer to the chroot(2)
system call or the chroot(8) wrapper program. The modified environment is called a
chroot jail.
What this means is that you can create a directory (for example /opt/root), copy
necessary program there and execute this program. For such program /opt/root/ will
be the root directory /. To understand why you would want that, read Wikipedia
chroot article.
It is practice time. You will do now create a minimal chroot environment with bash.
To do this you will create a directory structure and copy bash and its dependencies
into it.
Now you will learn how to create a chroot environment and enter into it.
Type this
1: sudo -s
2: ldd /bin/bash
3: mkdir -vp /opt/root/bin
4: mkdir -v /opt/root/lib
5: mkdir -v /opt/root/lib64
6: cp -v /bin/bash /opt/root/bin/
7: cp -v /lib/libncurses.so.5 /opt/root/lib/
8: cp -v /lib/libdl.so.2 /opt/root/lib
9: cp -v /lib/libc.so.6 /opt/root/lib
10: cp -v /lib64/ld-linux-x86-64.so.2 /opt/root/lib64
11: chroot /opt/root/
Wohoo, you just created yourself a Linux! Sort of.
What you should see
student@vm1:/opt~ sudo -s
root@vm1:/opt# ldd /bin/bash
linux-vdso.so.1 => (0x00007fff17bff000)
libncurses.so.5 => /lib/libncurses.so.5 (0x00007f4b1edc6000)
libdl.so.2 => /lib/libdl.so.2 (0x00007f4b1ebc2000)
libc.so.6 => /lib/libc.so.6 (0x00007f4b1e85f000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4b1f012000)
root@vm1:/opt# mkdir -vp /opt/root/bin
mkdir: created directory `/opt/root'
mkdir: created directory `/opt/root/bin'
root@vm1:/opt# mkdir -v /opt/root/lib
mkdir: created directory `/opt/root/lib'
root@vm1:/opt# mkdir -v /opt/root/lib64
mkdir: created directory `/opt/root/lib64'
root@vm1:/opt# cp -v /bin/bash /opt/root/bin/
`/bin/bash' -> `/opt/root/bin/bash'
root@vm1:/opt# cp -v /lib/libncurses.so.5 /opt/root/lib/
`/lib/libncurses.so.5' -> `/opt/root/lib/libncurses.so.5'
root@vm1:/opt# cp -v /lib/libdl.so.2 /opt/root/lib
75
`/lib/libdl.so.2' -> `/opt/root/lib/libdl.so.2'
root@vm1:/opt# cp -v /lib/libc.so.6 /opt/root/lib
`/lib/libc.so.6' -> `/opt/root/lib/libc.so.6'
root@vm1:/opt# cp -v /lib64/ld-linux-x86-64.so.2 /opt/root/lib64
`/lib64/ld-linux-x86-64.so.2' -> `/opt/root/lib64/ld-linux-x86-64.so.2'
root@vm1:/opt# chroot /opt/root/
Explanation
Executes bash as superuser (root).
Prints out libraries which bash needs to run.
Creates /opt/root/ and /opt/root/bin/ directories in one command. Neat, eh?
Creates /opt/root/lib directory.
Creates /opt/root/lib64 directory.
Copies /bin/bash to /opt/root/bin/.
Copies /lib/libncurses.so.5 to /opt/root/lib/.
Copies /lib/libdl.so.2 to /opt/root/lib/.
Copies /lib/libc.so.6 to /opt/root/lib/.
Copies /lib64/ld-linux-x86-64.so.2 to /opt/root/lib64/.
Changes root directory to /opt/root/.
Extra credit
Read man chroot, man ldd.
Copy ls command to your chroot and make it work.
A hard one: copy vim to your chroot and make it work.
Exercise 22. Filesystems: moving data
around: tar, dd
Time to see for yourself how everything in Linux is just a file.
This exercise is a big one, but its time to see what you have learned. After you will
make it through look up all purposefully unexplained program parameters in the
man, and try to explain to yourself what each command does.
Now you will learn how to juggle with data.
Do this
1: tar -czvf root.tgz /opt/root/
2: tar -tzvf root.tgz
3: cd /tmp
4: tar -zxvf ~/root.tgz
5: ls -al
6: dd_if=$(mount | grep /tmp | cut -d ' ' -f 1) && echo $dd_if
7: sudo dd if=$dd_if of=~/tmp.img bs=10M
8: cd && ls -alh
9: sudo losetup /dev/loop1 ~/tmp.img && sudo mount /dev/loop1 /mnt/
10: ls -al /mnt
11: sudo umount /mnt && sudo losetup -d /dev/loop1
12: sudo umount $dd_if && sudo mkfs.ext3 $dd_if
13: new_uuid=$(sudo tune2fs -l $dd_if | awk '/UUID/{print $3}') && echo $new_uuid
14: grep '/tmp' /etc/fstab
15: sed "s/^UUID=.*\/tmp\s\+ext3\s\+defaults\s\+[0-9]\s\+[0-9]\s\?/UUID=$new_uuid \/tmp ext3
defaults 0 2/" /etc/fstab
76
Now check the output using sudo tune2fs -l and sudo blkid. If replacement of
UUID in /etc/fstab looks sane, perform the actual replacement:
16: sudo sed -i'.bak' "s/^UUID=.*\/tmp\s\+ext3\s\+defaults\s\+[0-9]\s\+[0-9]\s\?/UUID=$new_uuid \/tmp
ext3 defaults 0 2/" /etc/fstab
17: sudo mount -a && ls /tmp
18: sudo umount /tmp && pv ~/tmp.img | sudo dd of=$dd_if bs=10M
19: new_uuid=$(sudo tune2fs -l $dd_if | awk '/UUID/{print $3}') && echo $new_uuid
20: sudo sed -i'.bak' "s/^UUID=.*\/tmp\s\+ext3\s\+defaults\s\+[0-9]\s\+[0-9]\s\?/UUID=$new_uuid \/tmp
ext3 defaults 0 2/" /etc/fstab
21: sudo mount -a
22: rm -v tmp.img
Type in y and press <ENTER>.
What you should see
student@vm1:~$ tar -czvf root.tgz /opt/root/
tar: Removing leading '/' from member names
/opt/root/
/opt/root/bin/
/opt/root/bin/bash
/opt/root/lib64/
/opt/root/lib64/ld-linux-x86-64.so.2
/opt/root/lib/
/opt/root/lib/libdl.so.2
/opt/root/lib/libncurses.so.5
/opt/root/lib/libc.so.6
student@vm1:~$ tar -tzvf root.tgz
drwxr-xr-x root/root
0 2012-07-05 03:14 opt/root/
drwxr-xr-x root/root
0 2012-07-05 03:14 opt/root/bin/
-rwxr-xr-x root/root 926536 2012-07-05 03:14 opt/root/bin/bash
drwxr-xr-x root/root
0 2012-07-05 03:14 opt/root/lib64/
-rwxr-xr-x root/root 128744 2012-07-05 03:14 opt/root/lib64/ld-linux-x86-64.so.2
drwxr-xr-x root/root
0 2012-07-05 03:14 opt/root/lib/
-rw-r--r-- root/root 14696 2012-07-05 03:14 opt/root/lib/libdl.so.2
-rw-r--r-- root/root 286776 2012-07-05 03:14 opt/root/lib/libncurses.so.5
-rwxr-xr-x root/root 1437064 2012-07-05 03:14 opt/root/lib/libc.so.6
student@vm1:~$ cd /tmp
student@vm1:/tmp$ tar -zxvf ~/root.tgz
opt/root/
opt/root/bin/
opt/root/bin/bash
opt/root/lib64/
opt/root/lib64/ld-linux-x86-64.so.2
opt/root/lib/
opt/root/lib/libdl.so.2
opt/root/lib/libncurses.so.5
opt/root/lib/libc.so.6
student@vm1:/tmp$ ls -al
total 19
drwxrwxrwt 6 root root 1024 Jul 5 04:17 .
drwxr-xr-x 22 root root 1024 Jul 3 08:29 ..
drwxrwxrwt 2 root root 1024 Jul 3 08:41 .ICE-unix
drwx------ 2 root root 12288 Jul 3 07:47 lost+found
drwxr-xr-x 3 student student 1024 Jul 5 03:24 opt
-rw-r--r-- 1 root root 489 Jul 3 10:14 sources.list
-r--r----- 1 root root 491 Jul 3 10:21 sudoers
drwxrwxrwt 2 root root 1024 Jul 3 08:41 .X11-unix
student@vm1:/tmp$ dd_if=$(mount | grep /tmp | cut -d ' ' -f 1) && echo $dd_if
77
/dev/sda8
student@vm1:~$ cd && ls -alh
total 243M
drwxr-xr-x 3 student student 4.0K Jul 5 04:27 .
drwxr-xr-x 4 root root 4.0K Jul 3 08:39 ..
-rw------- 1 student student 22 Jul 3 10:45 .bash_history
-rw-r--r-- 1 student student 220 Jul 3 08:39 .bash_logout
-rw-r--r-- 1 student student 3.2K Jul 3 08:39 .bashrc
-rw------- 1 student student 52 Jul 5 04:12 .lesshst
drwxr-xr-x 3 student student 4.0K Jul 5 03:23 opt
-rw-r--r-- 1 student student 675 Jul 3 08:39 .profile
-rw-r--r-- 1 student student 1.3M Jul 5 04:25 root.tgz
-rw-r--r-- 1 root root 241M Jul 5 04:36 tmp.img
student@vm1:~$ sudo losetup /dev/loop1 ~/tmp.img && sudo mount /dev/loop1 /mnt/
student@vm1:~$ ls -al /mnt
total 19
drwxrwxrwt 6 root root 1024 Jul 5 04:17 .
drwxr-xr-x 22 root root 1024 Jul 3 08:29 ..
drwxrwxrwt 2 root root 1024 Jul 3 08:41 .ICE-unix
drwx------ 2 root root 12288 Jul 3 07:47 lost+found
drwxr-xr-x 3 student student 1024 Jul 5 03:24 opt
-rw-r--r-- 1 root root 489 Jul 3 10:14 sources.list
-r--r----- 1 root root 491 Jul 3 10:21 sudoers
drwxrwxrwt 2 root root 1024 Jul 3 08:41 .X11-unix
student@vm1:~$ sudo umount /mnt && sudo losetup -d /dev/loop1
student@vm1:~$ sudo umount $dd_if && sudo mkfs.ext3 $dd_if
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
61752 inodes, 246784 blocks
12339 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
31 block groups
8192 blocks per group, 8192 fragments per group
1992 inodes per group
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729, 204801, 221185
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 27 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
student@vm1:~$ new_uuid=$(sudo tune2fs -l $dd_if | awk '/UUID/{print $3}') && echo $new_uuid
f8288adc-3ef9-4a6e-aab2-92624276b8ba
student@vm1:~$ grep '/tmp' /etc/fstab
# /tmp was on /dev/sda8 during installation
UUID=011b4530-e4a9-4d13-926b-48d9e33b64bf /tmp ext3 defaults 0 2
student@vm1:~$ sed "s/^UUID=.*\/tmp\s\+ext3\s\+defaults\s\+[0-9]\s\+[0-9]\s\?/UUID=$new_uuid \/tmp
ext3 defaults 0 2/" /etc/fstab
# /etc/fstab: static file system information.
#
78
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options>
<dump> <pass>
proc
/proc
proc defaults
0
0
# / was on /dev/sda1 during installation
UUID=91aacf33-0b35-474c-9c61-311e04b0bed1 /
ext3 errors=remount-ro 0
1
# /home was on /dev/sda9 during installation
UUID=e27b0efb-8cf0-439c-9ebe-d59c927dd590 /home
ext3 defaults
0
2
# /tmp was on /dev/sda8 during installation
UUID=f8288adc-3ef9-4a6e-aab2-92624276b8ba /tmp ext3 defaults 0 2
# /usr was on /dev/sda5 during installation
UUID=9f49821b-7f94-4915-b9a9-ed9f12bb6847 /usr
ext3 defaults
0
2
# /var was on /dev/sda6 during installation
UUID=b7e908a1-a1cd-4d5c-bc79-c3a99d003e7c /var
ext3 defaults
0
2
# swap was on /dev/sda7 during installation
UUID=292981d7-5a17-488f-8d9a-176b65f45d46 none
swap sw
0
0
/dev/scd0
/media/cdrom0 udf,iso9660 user,noauto 0
0
sudo sed -i'.bak' "s/^UUID=.*\/tmp\s\+ext3\s\+defaults\s\+[0-9]\s\+[0-9]\s\?/UUID=$new_uuid \/tmp ext3
defaults 0 2/" /etc/fstab
sudo mount -a && ls /tmp
student@vm1:~$ sudo umount /tmp && pv ~/tmp.img | sudo dd of=$dd_if bs=10M
241MB 0:00:04 [54.2MB/s]
[=========================================================================================
======================>] 100%
0+1928 records in
0+1928 records out
252706816 bytes (253 MB) copied, 5.52494 s, 45.7 MB/s
student@vm1:~$ rm -v tmp.img
rm: remove write-protected regular file `tmp.img'? y
removed `tmp.img'
student@vm1:~$
Explanation
Creates in archive or /opt/root/ in your home directory. Archive file has extension
.tgz because this archive actually consists of two parts, like a matryoshka doll. First
part is designated by letter t and is one big file in which all archived files are merged
by program tar. Second part is designated by letters gz and means that tar called
gzip program for you to compress it.
Tests this archive.
Changes directory to /tmp.
Extracts your archive.
Prints out directory contents.
Extracts name of a partition which is mounted on /tmp, stores it in dd_if variable,
and if extraction was successful prints out dd_if value. if stands for input file.
Copies whole partition to tmp.img in your home directory. dd is called as superuser
because you are accessing file /dev/sda8 representing your partition which is not
accessible for normal users.
Changes directory to your home directory and prints out its content.
Tells Linux to use tmp.img file as a physical partition (sort of) and mounts it.
Prints out content of tmp.img. You are able to see that it really is exact copy of /tmp.
Unmounts tmp.img and tells Linux to stop treating is as partition.
79
Unmounts /tmp and creates new filesystem there, deleting everything which was
there in the process.
Extracts UUID of your new /tmp filesystem, stores it in new_uuid and prints it out.
Prints out a line describing old /tmp partition from /etc/fstab.
Shows you how modified /etc/fstab will look. It is done by using regular expression,
which work as a mask which define this line:
UUID=f8288adc-3ef9-4a6e-aab2-92624276b8ba /tmp ext3 defaults 0 2
After you finish this book I will give you a link which will allow you to learn how to
create such regular expressions.
Makes actual replacement of /tmp old UUID with new UUID.
Mounts all filesystems described in /etc/fstab and lists contents of your new /tmp
Unmounts new /tmp and restores old /tmp from tmp.img.
Gets old /tmp UUID, which is actually the same as it was before you created a new
filesystem, because tmp.img is perfect copy of your old /tmp.
Replaces new UUID with old UUID in your /etc/fstab.
Mounts all filesystems from /etc/fstab. If this command does not result in error,
chances are you did everything right. Congratulations.
Removes tmp.img from your home directory.
Extra credit
Try to explain in detail what each command does. Take a list of paper and write it all
out. Look up all not well understood commands and parameters in the man.
It is a bit early for this, but why you were able to remove tmp.img from your home
directory issuing removal command as student, considering that tmp.img was
created as root?
80
Exercise 23. Filesystems: security
permissions, chown, chmod, umask
It is time to understand Linux filesystems security model. Let us start with quoting
Wikipedia article on permissions:
Most current file systems have methods of administering permissions or access
rights to specific users and groups of users. These systems control the ability of the
users to view or make changes to the contents of the filesystem.
Permissions on Unix-like systems are managed in three distinct classes. These
classes are known as user, group, and others. In effect, Unix permissions are a
simplified form of access control lists (ACLs).
When a new file is created on a Unix-like system, its permissions are determined
from the umask of the process that created it.
For each file in Linux there are three permission classes associated with it. For each
permission class there are three permissions.
This are permission classes:
Class Description
user One system user. The file is said to be owned by this user.
group One group of users
others Any other users or groups
This are permissions, assignable for each class:
Symbolic
Permission
Description
notation
r-read
Ability to read the file
-wwrite
Ability to write to the file
Ability to execute file as a program,
--x
execute
for example a shell script should have this set
This two tables can be summed up:
Owner Group Others
r w x r w x r w
x
This permissions are represented as numbers. Consider the following output:
student@vm1:~$ ls -al tmp.img
-rw-r--r-- 1 root root 252706816 Jul 6 07:54 tmp.img
student@vm1:~$ stat tmp.img
File: 'tmp.img'
Size: 252706816
Blocks: 494064 IO Block: 4096 regular file
Device: 809h/2057d
Inode: 88534
Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2012-07-06 07:56:58.000000000 -0400
Modify: 2012-07-06 07:54:54.000000000 -0400
Change: 2012-07-06 07:54:54.000000000 -0400
student@vm1:~$
Here we are able to see that tmp.img is owned by user root and group root and has
the following permissions: -rw-r–r–. Let us learn to read them.
-rw
r-r-1
# Owner can read and write to the file
# Group can only read the file
# Others can also only read the file
#
81
root
# Owner is user root
root
# Group is root (but remember not to confuse this with user)
252706816 #
Jul
#
6
#
07:54
#
tmp.img #
Here are the same permissions is octal notation:
Access:
(
0
6 -rw
4 r-4 r-)
Uid: ( 0/
Gid: ( 0/
root)
root)
And this is the table for translating from octal to symbolic notation:
Symbolic Octal Binary . Symbolic Octal Binary
--0
000
4
101
. r---x
1
001
5
100
. r-x
-w2
010
6
110
. rw-wx
3
011
7
111
. rwx
Notice that resulting permissions are obtained with simple addition. For example, let
us take r-x permission. r in octal notation is 4, x is 1, 1+4 is 5 which is r-x.
Now lets talk about zero in stat output 0644. This is for setting up something called
SUID, SGID and Sticky bit. I will not cover it in detail, but I will give you an Extra
Credit task for this and translation tables.
Special bits:
Symbolic
Mode
Description
notataion
SUID u-Set User ID upon execution.
SGID -gSet Group ID upon execution.
Works only for directories, when set, files in that directory may only
Sticky --s
be unlinked or renamed by root or their owner.
Translating special bits from symbolic to octal notation:
Symbolic Octal Binary . Symbolic Octal Binary
--0
000
4
101
. u---s
1
001
5
100
. u-s
-g2
010
6
110
. ug-gs
3
011
7
111
. ugs
Now what about newly created file? For example, you created a file with touch
umask.test, which permissions will it have? It turns out that you are able to control
this with file mode creation mask, umask. Is a mechanism to define which
permissions assign to files when you are creating it. The umask works by masking
out, that is substracting permissions from default ones, which for bash are 777 for
directories and 666 for files. Umask is define for user, group and others also.
Mapping betweend umask values and permissions:
Symbolic Octal Binary . Symbolic Octal Binary
82
4
101
. -wc
rw1
001
5
100
. -wr-x
2
010
6
110
. --x
r-3
011
7
111
. --To understand more clearly, here is another table. Remember that this permissions
are masked out, that is they are removed. For simplicity in this example user, group
and others permissions are the same.
Effective
permissions
Umask
Masked out (removed)
for a new file with Notes
value
permisssions
default
permissions 666
666 write, read and All default permissions
000
none
execute
are preserved
666 write, read and Because new files are
111
execute only
execute
not executable
555 read and
222
write only
execute
333
write and execute
444 read only
333 write and
444
read only
execute
555
read and execute
222 write
666
read and write
111 execute
No permissions are
777
read, write and execute
000 nothing
preserved
Another umask example:
Octal Symbolic
022
--- -w- -wumask
New files
666
rw- rw- rwInitial file permission
022
--- -w- -wCompliment of umask
644
rw- r-- r-Resultant file permission
New directory
777
rwx rwx rwx
Initial directory permission
022
--- -w- -wComplement of umask
rwx r-x r-x
Resultant directory permission 655
Let us summarize this infodump:
Permissions, or access rights — mechanism for controlling access to files and
directories.
Permission modes — types of permissions which allow actions with files.
Read, r — ability to read the file.
Write, w — ability to write to the file.
Execute, x — ability to execute the file as a program. For directories this has a
special meaning, namely it allows directory to be listed.
Classes of users — entities to which permissions are applied.
rwx
0
000
83
User/owner class, u — owner of file or directory, often is is the one who created
them.
Group class, g — group is a collections of users.
Others class, o — everyone else except owner and group.
Umask — a mechanism for controlling access to newly created files.
And the commands to manage permissions:
chmod — change file permissions.
chown — change owner permissions.
umask — change mask for assigning permissions to new files.
Now you will learn how to change file permissions, file owner and umask.
Do this
1: umask
2: echo 'test' > perms.022
3: ls -l perms.022
4: stat perms.022 | grep 'Access: ('
5: chmod 000 perms.022
6: ls -al perms.0022
7: echo 'test' > perms.022
8: rm -v perms.022
Remember question from Extra Credit of previous exercise? You are in similar
situation now, because you are not allowed to do anything with this file. But why you
are allowed to remove it? That is because when removing file, you are actually
removing information about this file from directory, doing nothing with file itself. I
have nice extra credit for you on this topic.
9: umask 666
10: echo 'test' > perms.000
11: ls -l perms.000
12: cat perms.000
13: chmod 600 perms.000
14: cat perms.000
15: rm -v perms.000
16: umask 027
17: echo 'test' > perms.027
18: ls -l perms.027
19: sudo chown root perms.027
20: echo 'test1' >> perms.027
21: chown student perms.027
22: sudo chown student perms.027
23: echo 'test1' >> perms.027
24: rm -v perms.027
25: umask 022
What you should see
student@vm1:~$ umask
0027
student@vm1:~$ echo 'test' > perms.022
student@vm1:~$ ls -l perms.022
-rw-r----- 1 student student 5 Jul 9 10:23 perms.022
student@vm1:~$ stat perms.022 | grep 'Access: ('
Access: (0640/-rw-r-----) Uid: ( 1000/ student) Gid: ( 1000/ student)
student@vm1:~$ chmod 000 perms.022
student@vm1:~$ ls -al perms.0022
ls: cannot access perms.0022: No such file or directory
student@vm1:~$ echo 'test' > perms.022
-bash: perms.022: Permission denied
84
student@vm1:~$ rm -v perms.022
rm: remove write-protected regular file `perms.022'? y
removed `perms.022'
student@vm1:~$ umask 666
student@vm1:~$ echo 'test' > perms.000
student@vm1:~$ ls -l perms.000
---------- 1 student student 5 Jul 9 10:23 perms.000
student@vm1:~$ cat perms.000
cat: perms.000: Permission denied
student@vm1:~$ chmod 600 perms.000
student@vm1:~$ cat perms.000
test
student@vm1:~$ rm -v perms.000
removed `perms.000'
student@vm1:~$ umask 027
student@vm1:~$ echo 'test' > perms.027
student@vm1:~$ ls -l perms.027
-rw-r----- 1 student student 5 Jul 9 10:24 perms.027
student@vm1:~$ sudo chown root perms.027
student@vm1:~$ echo 'test1' >> perms.027
-bash: perms.027: Permission denied
student@vm1:~$ chown student perms.027
chown: changing ownership of `perms.027': Operation not permitted
student@vm1:~$ sudo chown student perms.027
student@vm1:~$ echo 'test1' >> perms.027
student@vm1:~$ rm -v perms.027
removed `perms.027'
student@vm1:~$ umask 022
Explanation
Prints current umask.
Creates file perms.022 containing line test.
Prints out information about this file.
Prints out permission information about this file in octal notation.
Changes permissions on this file, forbidding anyone to do anything with it.
Prints out information about this file.
Tries to replace this file contents with line 'test', failing because of absent
permissions.
Removes this file. This is is possible because file itself is not being touched, only an
entry from directory /home/student is.
Changes umask to assign none permissions by default.
Creates file perms.000 containing line test.
Prints out information about this file.
Tries to print out this file content, which obviously results in error.
Changes file permissions to allow owner to read and write it.
Prints this file contents, this time successfully.
Removes this file.
Changes umask once more.
Creates file perms.027 containing line test.
Prints out information about this file.
Changes file owner to root.
Tries to append line test1 to this file, what results in error.
Tries to change file owner back to student, failing this because information about file
owner is contained in file itself, more precisely in its index node.
85
Changes file owner back to student, this time succeeding because run as root.
Adds line test1 to our file, this time successfully.
Removes perms.027.
Returns umask to its default value.
Extra credit
Read man chmod, man chown, man umask.
Reread man chmod on setuid, setgid and sticky bits. Set your directory setuid bit
in such a way that when doing umask 002 && echo test | sudo tee perms.root
student was the resulting group of perms.root.
Find out why umask 002 did not work.
Try this:
student_block0=$(echo 'stat /student' | sudo debugfs /dev/sda9 2>/dev/null | grep '(0)' | cut -d':' -f2)
echo $student_block0
sudo dd if=/dev/sda9 bs=4096 skip=$student_block0 count=1 | hexdump -C
Cool, huh? You have just read the directory contents directly from raw partition. Well,
when you are deleting a file, an entry is deleted from here, and you have
permissions to modify this entries because this is what directory (a special file)
actually is.
86
Exercise 24. Networking: interface
configuration, ifconfig, netstat,
iproute2, ss, route
This exercise is a big one in terms of info dumped on you, and if you are not familiar
with networking this is gonna hurt. If you feel absolutely lost, skip to the Do This part
right away and be done with it. To understand this part properly you should be at
least superfluously familiar with the following basic concepts of networking:
Communications protocol — a communications protocol which is a system of
digital message formats and rules for exchanging those messages in or between
computing systems and in telecommunications.
Ethernet — family of computer networking technologies for local area networks
(LANs).
MAC address — an unique identifier assigned to network interfaces for
communications on the physical network segment. Example: 08:00:27:d4:45:68.
TCP/IP — the Internet protocol suite is the set of communications protocols used for
the Internet and similar networks, and generally the most popular protocol stack for
wide area networks. It is commonly known as TCP/IP, because of its most important
protocols: Transmission Control Protocol (TCP) and Internet Protocol (IP)
IP — the Internet Protocol (IP) is the principal communications protocol used for
relaying datagrams (also known as network packets) across an internetwork using
the Internet Protocol Suite.
IP address — an Internet Protocol address. Example: 10.0.2.15
Port — application-specific or process-specific software construct serving as a
communications endpoint in a computer's host operating system. Example: 22
Network socket — an endpoint of an inter-process communication flow across a
computer network. Today, most communication between computers is based on the
Internet Protocol; therefore most network sockets are Internet sockets.
Local socket address — local IP address and port number, example: 10.0.2.15:22.
Remote socket address — remote IP address and port number, only for
established TCP sockets. Example: 10.0.2.2:52173.
Socket pair — communicating local and remote sockets, only TCP protocol.
Example: (10.0.2.15:22, 10.0.2.2:52173).
Subnet mask — logically visible subdivision of an IP network. Example: /24 or, in
another notation, 255.255.255.0.
Routing — the process of selecting paths in a network along which to send network
traffic.
Default gateway — in computer networking, a gateway is a node (a router) on a
TCP/IP network that serves as an access point to another network. A default
gateway is the node on the computer network that the network software uses when
an IP address does not match any other routes in the routing table. Example:
10.0.2.2.
Broadcast address — logical address at which all devices connected to a multipleaccess communications network are enabled to receive datagrams. A message sent
to a broadcast address is typically received by all network-attached hosts, rather
than by a specific host. Example: 10.0.2.255.
ICMP — Internet Control Message Protocol, example usage: ping 10.0.2.2.
87
TCP — Transmission Control Protocol. Establishes connection before data can be
exchanged, therefore reliable by design. Example users: SSH, HTTP.
UDP — User Datagram Protocol. Transfers data without establishing connection,
therefore unreliable by design. Example users: DNS.
If some of this concepts are not familiar to you, no worries.
Read corresponding Wikipedia articles until you achieve at least superfluous
understanding (but good old hardcore grinding is of course better).
Watch this videos from http://www.visualland.net.cn/:
Expand IP Address tree node in the left of the site and work you way through it.
Expand TCP tree node and do the same.
Read Linux networking concepts introduction. This guide is good because it even
acknowledges that The Internet is for porn.
Let us continue. This is list of Linux network-related commands:
ifconfig — configure and view status of network interface.
netstat — print network connections, routing tables, interface statistics, masquerade
connections, and multicast memberships.
ip — show / manipulate routing, devices, policy routing and tunnels.
ss — another utility to investigate sockets.
Now let us examine what information each command can show us. We will start with
ifconfig.
student@vm1:~$ sudo ifconfig
eth0
Link encap:Ethernet HWaddr 08:00:27:d4:45:68
# (1), (2), (3)
inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0 # (4), (5), (6), (7)
inet6 addr: fe80::a00:27ff:fed4:4568/64 Scope:Link
# (8), (9), (10)
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
# (11), (12), (13), (14),
(15), (16)
RX packets:35033 errors:0 dropped:0 overruns:0 frame:0
# (17), (18), (19), (20), (21), (22)
TX packets:28590 errors:0 dropped:0 overruns:0 carrier:0 # (23), (24), (25), (26), (27), (28)
collisions:0 txqueuelen:1000
# (29), (30)
RX bytes:6360747 (6.0 MiB) TX bytes:21721365 (20.7 MiB) # (31), (32)
lo
Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:560 (560.0 B) TX bytes:560 (560.0 B)
We can see that there are two network interfaces in vm1, eth0 and lo. lo is a
loopback interface, which is used to connect to client-server programs on the same
machine.
Let us see what information we have on eth0, which is a VirtualBox pseudo network
interface:
Field
Description
Field
Description
(1) Link
Physical options
(17) RX
Receive (abbreviation)
(2) encap
Encapsulation type
(18) packets Total number of packets
Total number of packets
(3) Hwaddr
MAC-address
(19) errors
with errors
Dropped packets (low
(4) inet
Address family (IPv4)
(20) dropped
system memory?)
88
(5) addr
IPv4 address
(6) Bcast
(7) Mask
(8) inet6
Broadcast address
Network mask
Address family (IPv6)
(9) addr
IPv6 address
(10) Scope
Address scope (host,
link, global)
(11) UP
Interface is functioning
(21)
overruns
(22) frame
(23) TX
(24) packets
Packets come faster than
they can be processed
Invalid frames received
Transmit (abbreviation)
Total number of packets
Total number of packets
(25) errors
with errors
Dropped packets (low
(26) dropped
system memory?)
(27)
An app sends packes very
overruns
fast? (I am not sure)
(12)
BROADCAST
It can send traffic to all
(28) carrier Loss of link carrier wave
hosts at once
It is ready to accept data (29)
(13) RUNNING
Packed collision did occur
(I am not sure)
collisions
(14)
It can send and receive (30)
Transmin queue length for
MULTICAST
multicast packets
txqueuelen outgoing packets
Its Maximum transfer
(31) RX
(15) MTU
Total received bytes
unit
bytes
Route cost (not used in (32) TX
(16) Metric
Total sent bytes
Linux)
bytes
That sure is a lot. But then again, for now only important fields are:
(5) addr — IPv4 address.
(6) Bcast — Broadcast address.
(7) Mask — Network mask.
(11) UP — Interface is functioning.
(13) RUNNING — It is ready to accept data.
(19) errors and (25) errors — if there is something different from zero here, we have
problems.
Now let us see what netstat can show us.
student@vm1:~$ sudo netstat -ap
Active Internet connections (servers and established)
#(1) (2) (3) (4)
(5)
(6)
(7)
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address
Foreign Address
State
PID/Program name
tcp
0
0 *:sunrpc
*:*
LISTEN
580/portmap
tcp
0
0 *:ssh
*:*
LISTEN
900/sshd
tcp
0
0 localhost:smtp
*:*
LISTEN
1111/exim4
tcp
0
0 *:36286
*:*
LISTEN
610/rpc.statd
tcp
0
0 10.0.2.15:ssh
10.0.2.2:52191
ESTABLISHED 12023/sshd: student [
tcp
0
0 10.0.2.15:ssh
10.0.2.2:48663
ESTABLISHED 11792/sshd: student [
tcp6
0
0 [::]:ssh
[::]:*
LISTEN
900/sshd
tcp6
0
0 ip6-localhost:smtp
[::]:*
LISTEN
1111/exim4
udp
0
0 *:bootpc
*:*
843/dhclient
udp
0
0 *:sunrpc
*:*
580/portmap
udp
0
0 *:52104
*:*
610/rpc.statd
udp
0
0 *:786
*:*
610/rpc.statd
#(8) (9) (10)
(11)
(12)
(13) (14)
(15)
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags
Type
State
I-Node PID/Program name Path
unix 2
[ ACC ] STREAM LISTENING 3452 786/acpid
/var/run/acpid.socket
89
unix
unix
unix
unix
unix
unix
unix
unix
unix
unix
unix
unix
unix
6
2
2
3
3
2
3
3
2
2
2
3
3
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
DGRAM
DGRAM
DGRAM
STREAM
STREAM
DGRAM
STREAM
STREAM
DGRAM
DGRAM
DGRAM
DGRAM
DGRAM
3407
1940
88528
CONNECTED
CONNECTED
68563
CONNECTED
CONNECTED
66680
3465
3448
1945
1944
751/rsyslogd
/dev/log
214/udevd
@/org/kernel/udev/udevd
30939/sudo
68565 12023/sshd: student [
68564 12026/1
12023/sshd: student [
66682 11792/sshd: student [
66681 11794/0
11792/sshd: student [
843/dhclient
786/acpid
214/udevd
214/udevd
I used two parameters to modify netstat output. -a parameter told netstat to show us
all connections, both established, like your current ssh session through which your
are typing, and listening, like sshd daemon waiting for new connections. -p told
netstat to show which program owns each connection.
Active Internet Connections (servers and established)
Field
Description
(1) Proto
The protocol (tcp, udp, raw) used by the socket.
The count of bytes not copied by the user program connected to this
(2) Recv-Q
socket.
(3) Send-Q The count of bytes not acknowledged by the remote host.
(4) Local
Address and port number of the local end of the socket.
Address
(5) Foreign
Address and port number of the remote end of the socket.
Address
ESTABLISHED, SYN_SENT, SYN_RECV, FIN_WAIT1, FIN_WAIT2,
(6) State
TIME_WAIT, CLOSE, CLOSE_WAIT, LAST_ACK, LISTEN,
CLOSING, UNKNOWN
Slash-separated pair of the process id (PID) and process name of the
(7) PID
process that owns the socket.
Active UNIX domain sockets (servers and established)
Field
Description
(8) Proto
The protocol (usually unix) used by the socket.
(9) RefCnt
The reference count (i.e. attached processes via this socket).
The flags displayed is SO_ACCEPTON (displayed as ACC),
(10) Flags
SO_WAITDATA (W) or SO_NOSPACE (N).
SOCK_DGRAM, SOCK_STREAM, SOCK_RAW, SOCK_RDM,
(11) Type
SOCK_SEQPACKET, SOCK_PACKET, UNKNOWN.
FREE, LISTENING, CONNECTING, CONNECTED,
(12) State
DISCONNECTING, (empty), UNKNOWN.
(13) I-Node I-Node of socket file.
Process ID (PID) and process name of the process that has the socket
(14) PID
open.
This is the path name as which the corresponding processes attached
(15) Path
to the socket.
90
Not all fields are actually important. Usually you have to look only in Active Internet
Connections (servers and established) section, and use this fields:
(1) Proto — The protocol (tcp, udp, raw) used by the socket.
(4) Local Address — address and port number of the local end of the socket.
(5) Foreign Address — address and port number of the remote end of the socket,
only for socket pairs.
(6) State — for now you should know only two states: LISTEN and ESTABLISHED.
First means that you may connect to this socket, second means that this socket
already is connected, and in this cases netstat shows you socket pair.
ip is a program similar to ifconfig with extended capabilities. It is from iproute2 suite
which is intended to replace ifconfig some day. Example output:
student@vm1:~$ sudo ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
#
(1)
(2)
(3) (4)
(5)
(6)
(8)
(9)
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:d4:45:68 brd ff:ff:ff:ff:ff:ff
# (9), (10), (11)
inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
# (12), (13), (14)
inet6 fe80::a00:27ff:fed4:4568/64 scope link
# (15), (16)
valid_lft forever preferred_lft forever
# (17), (18), (19)
Again, let us see what information we have on eth0:
Field
Description
(1) BROADCAST It can send traffic to all hosts at once
(2) MULTICAST It can send and receive multicast packets
(3) UP
It is functioning, logical state
(4) LOWER_UP Driver signals L1 up (since Linux 2.6.17)
(5) MTU
Maximum transfer unit
(6) qdisc
Queueing discipline, basically traffic scheduler policy
(8) State
Physical state (carrier sense?)
(9) qlen
Transmin queue length for outgoing packets
(10) link
Physical options
(11) ether
Encapsulation type, MAC-address
(12) brd
Data Link Layer (physical) broadcast address
(13) inet
IPv4 address family address
(14) brd
IPv4 boradcast
(15) scope
IPv4 Address scope (host, link, global)
(16) inet6
IPv6 address family address
(17) scope
IPv6 Address scope (host, link, global
(18) valid_lft
IPv6 source address selection policy
(19) preffered_lft IPv6 source address selection policy
You already know which parameters are important (the same as in ifconfig).
ss is basically contemporary netstat with extended capabilities. This is its example
output, interpretation of which is left as exercise for you:
student@vm1:~$ sudo ss -ap | cut -c1-200
State
Recv-Q Send-Q Local Address:Port
Peer Address:Port
91
LISTEN 0
128
*:sunrpc
LISTEN 0
128
:::ssh
LISTEN 0
128
*:ssh
LISTEN 0
20
::1:smtp
LISTEN 0
20
127.0.0.1:smtp
LISTEN 0
128
*:36286
ESTAB
0
0
10.0.2.15:ssh
users:(("sshd",12023,3),("sshd",12026,3))
ESTAB
0
0
10.0.2.15:ssh
users:(("sshd",11792,3),("sshd",11794,3))
*:*
users:(("portmap",580,5))
:::*
users:(("sshd",900,4))
*:*
users:(("sshd",900,3))
:::*
users:(("exim4",1111,4))
*:*
users:(("exim4",1111,3))
*:*
users:(("rpc.statd",610,7))
10.0.2.2:52191
10.0.2.2:48663
This is for working with interface, connetctions and interface addresses. But what
about network routes? You can get information about them using several commands
as well:
student@vm1:~$ sudo route -n
Kernel IP routing table
# (1)
(2)
(3)
(4) (5) (6) (7) (8)
Destination Gateway
Genmask
Flags Metric Ref Use Iface
10.0.2.0
*
255.255.255.0 U 0
0
0 eth0
default
10.0.2.2
0.0.0.0
UG 0
0
0 eth0
student@vm1:~$ sudo netstat -nr
Kernel IP routing table #
(9)
Destination Gateway
Genmask
Flags MSS Window irtt Iface
10.0.2.0
0.0.0.0
255.255.255.0 U
00
0 eth0
0.0.0.0
10.0.2.2
0.0.0.0
UG
00
0 eth0
student@vm1:~$ sudo ip route show
# (10)
(11)
(12)
(13)
(14)
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
#(15)
(16)
default via 10.0.2.2 dev eth0
Let us work through fields onece more:
Field
Description
(1)
The destination network or destination host.
Destination
(2) Gateway The gateway address or '*' if none set.
The netmask for the destination net; '255.255.255.255' for a host
(3) Genmask
destination and '0.0.0.0' for the default route.
Up, Host, Gateway, Reinstate, Dynamically installed, Modified,
(4) Flags
Addrconf, Cache entry, ! reject.
The 'distance' to the target (usually counted in hops). It is not used by
(5) Metric
recent kernels, but may be needed by routing daemons.
(6) Ref
Number of references to this route. (Not used in the Linux kernel.)
(7) Use
Count of lookups for the route.
(8) Iface
Interface to which packets for this route will be sent.
Initial RTT (Round Trip Time). The kernel uses this to guess about the
(9) irtt
best TCP protocol parameters
without waiting on (possibly slow) answers.
(10)
The destination network or destination host.
Net/Mask
(11) dev
Interface to which packets for this route will be sent.
(12) proto
man ip /RTPROTO: redirect, kernel, boot, static, ra
(13) scope man ip /SCOPE_WALUE: global, site, link, host
92
The source address to prefer when sending to the destinations
covered by the route prefix.
Gateway address for all addresses without explicitly assigned
(15) default
gateways
(15) dev
Interface to which packets for this route will be sent.
The important field for now:
(1) Destination — The destination network or destination host.
(2) Gateway — The gateway address or '*' if none set. Default means that packets
will be sent via this gateway if there is no explicitly specified gateway for packet
destination address.
(3) Genmask — The netmask for the destination net; '255.255.255.255' for a host
destination and '0.0.0.0' for the default route.
(8) Iface — Interface to which packets for this route will be sent.
Which field from netstat and route correspond to which from ip route show is left
as an exercise for you once more. Well, that was awfully big! Take a deep breath
and let us move to the practice.
Now you will learn how to create pseudo-interface, assign address to it and change it
state.
(14) src
Do this
1: sudo aptitude install uml-utilities
2: sudo tunctl -t tap0 -u student
3: ls -al /sys/devices/virtual/net/tap0
4: sudo ifconfig tap0 10.1.1.1 netmask 255.255.255.0
5: sudo ifconfig
6: sudo route
7: ping 10.1.1.1 -c 2
8: sudo ifconfig tap0 down
9: ping 10.1.1.1 -c 2
10: sudo ifconfig tap0 up
11: sudo ip a a 10.2.2.2/24 brd + dev tap0
12: sudo ifconfig
13: sudo route
14: ip a s
15: ip r s
16: ping 10.2.2.2 -c 2
17: sudo ip link set dev tap0 down
18: ip a s
19: ip r s
20: ping 10.2.2.2 -c 2
21: sudo tunctl -d tap0
22: ip a s
23: ls -al /sys/devices/virtual/net/tap0
What you should see
student@vm1:~$ sudo aptitude install uml-utilities
The following NEW packages will be installed:
libfuse2{a} uml-utilities
0 packages upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/205 kB of archives. After unpacking 737 kB will be used.
Do you want to continue? [Y/n/?]
Selecting previously deselected package libfuse2.
(Reading database ... 39616 files and directories currently installed.)
Unpacking libfuse2 (from .../libfuse2_2.8.4-1.1_amd64.deb) ...
Selecting previously deselected package uml-utilities.
Unpacking uml-utilities (from .../uml-utilities_20070815-1.1_amd64.deb) ...
93
Processing triggers for man-db ...
Setting up libfuse2 (2.8.4-1.1) ...
Setting up uml-utilities (20070815-1.1) ...
Starting User-mode networking switch: uml_switch.
student@vm1:~$ sudo tunctl -t tap0 -u student
Set 'tap0' persistent and owned by uid 1000
student@vm1:~$ ls -al /sys/devices/virtual/net/tap0
total 0
drwxr-xr-x 4 root root 0 Jul 11 05:33 .
drwxr-xr-x 4 root root 0 Jul 11 05:33 ..
-r--r--r-- 1 root root 4096 Jul 11 05:33 address
-r--r--r-- 1 root root 4096 Jul 11 05:33 addr_len
-r--r--r-- 1 root root 4096 Jul 11 05:33 broadcast
-r--r--r-- 1 root root 4096 Jul 11 05:33 carrier
-r--r--r-- 1 root root 4096 Jul 11 05:33 dev_id
-r--r--r-- 1 root root 4096 Jul 11 05:33 dormant
-r--r--r-- 1 root root 4096 Jul 11 05:33 duplex
-r--r--r-- 1 root root 4096 Jul 11 05:33 features
-rw-r--r-- 1 root root 4096 Jul 11 05:33 flags
-r--r--r-- 1 root root 4096 Jul 11 05:33 group
-rw-r--r-- 1 root root 4096 Jul 11 05:33 ifalias
-r--r--r-- 1 root root 4096 Jul 11 05:33 ifindex
-r--r--r-- 1 root root 4096 Jul 11 05:33 iflink
-r--r--r-- 1 root root 4096 Jul 11 05:33 link_mode
-rw-r--r-- 1 root root 4096 Jul 11 05:33 mtu
-r--r--r-- 1 root root 4096 Jul 11 05:33 operstate
-r--r--r-- 1 root root 4096 Jul 11 05:33 owner
drwxr-xr-x 2 root root 0 Jul 11 05:33 power
-r--r--r-- 1 root root 4096 Jul 11 05:33 speed
drwxr-xr-x 2 root root 0 Jul 11 05:33 statistics
lrwxrwxrwx 1 root root 0 Jul 11 05:33 subsystem -> ../../../../class/net
-r--r--r-- 1 root root 4096 Jul 11 05:33 tun_flags
-rw-r--r-- 1 root root 4096 Jul 11 05:33 tx_queue_len
-r--r--r-- 1 root root 4096 Jul 11 05:33 type
-rw-r--r-- 1 root root 4096 Jul 11 05:33 uevent
student@vm1:~$ sudo ifconfig tap0 10.1.1.1 netmask 255.255.255.0
student@vm1:~$ sudo ifconfig
eth0
Link encap:Ethernet HWaddr 08:00:27:d4:45:68
inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fed4:4568/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:64040 errors:0 dropped:0 overruns:0 frame:0
TX packets:44578 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:19663646 (18.7 MiB) TX bytes:25043918 (23.8 MiB)
lo
Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:76 errors:0 dropped:0 overruns:0 frame:0
TX packets:76 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:6272 (6.1 KiB) TX bytes:6272 (6.1 KiB)
tap0
Link encap:Ethernet HWaddr ee:d8:2e:f6:bc:f1
inet addr:10.1.1.1 Bcast:10.1.1.255 Mask:255.255.255.0
inet6 addr: fe80::ecd8:2eff:fef6:bcf1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
94
TX packets:0 errors:0 dropped:1 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
student@vm1:~$ sudo route
Kernel IP routing table
Destination Gateway
Genmask
Flags Metric Ref Use Iface
10.0.2.0
*
255.255.255.0 U 0
0
0 eth0
10.1.1.0
*
255.255.255.0 U 0
0
0 tap0
default
10.0.2.2
0.0.0.0
UG 0
0
0 eth0
student@vm1:~$ ping 10.1.1.1 -c 2
PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
64 bytes from 10.1.1.1: icmp_req=1 ttl=64 time=0.070 ms
64 bytes from 10.1.1.1: icmp_req=2 ttl=64 time=0.027 ms
--- 10.1.1.1 ping statistics --2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.027/0.048/0.070/0.022 ms
student@vm1:~$ sudo ifconfig tap0 down
student@vm1:~$ ping 10.1.1.1 -c 2
PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
64 bytes from 10.1.1.1: icmp_req=1 ttl=64 time=0.030 ms
64 bytes from 10.1.1.1: icmp_req=2 ttl=64 time=0.024 ms
--- 10.1.1.1 ping statistics --2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.024/0.027/0.030/0.003 ms
student@vm1:~$ sudo ifconfig tap0 up
student@vm1:~$ sudo ip a a 10.2.2.2/24 brd + dev tap0
student@vm1:~$ sudo ifconfig
eth0
Link encap:Ethernet HWaddr 08:00:27:d4:45:68
inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fed4:4568/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:64088 errors:0 dropped:0 overruns:0 frame:0
TX packets:44609 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:19667480 (18.7 MiB) TX bytes:25049771 (23.8 MiB)
lo
Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:84 errors:0 dropped:0 overruns:0 frame:0
TX packets:84 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:6944 (6.7 KiB) TX bytes:6944 (6.7 KiB)
tap0
Link encap:Ethernet HWaddr ee:d8:2e:f6:bc:f1
inet addr:10.1.1.1 Bcast:10.1.1.255 Mask:255.255.255.0
inet6 addr: fe80::ecd8:2eff:fef6:bcf1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:9 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
student@vm1:~$ sudo route
Kernel IP routing table
Destination Gateway
Genmask
Flags Metric Ref Use Iface
10.2.2.0
*
255.255.255.0 U 0
0
0 tap0
95
10.0.2.0
*
255.255.255.0 U 0
0
0 eth0
10.1.1.0
*
255.255.255.0 U 0
0
0 tap0
default
10.0.2.2
0.0.0.0
UG 0
0
0 eth0
student@vm1:~$ ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:d4:45:68 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
inet6 fe80::a00:27ff:fed4:4568/64 scope link
valid_lft forever preferred_lft forever
12: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN
qlen 500
link/ether ee:d8:2e:f6:bc:f1 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.1/24 brd 10.1.1.255 scope global tap0
inet 10.2.2.2/24 brd 10.2.2.255 scope global tap0
inet6 fe80::ecd8:2eff:fef6:bcf1/64 scope link
valid_lft forever preferred_lft forever
student@vm1:~$ ip r s
10.2.2.0/24 dev tap0 proto kernel scope link src 10.2.2.2
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
10.1.1.0/24 dev tap0 proto kernel scope link src 10.1.1.1
default via 10.0.2.2 dev eth0
student@vm1:~$ ping 10.2.2.2 -c 2
PING 10.2.2.2 (10.2.2.2) 56(84) bytes of data.
64 bytes from 10.2.2.2: icmp_req=1 ttl=64 time=0.081 ms
64 bytes from 10.2.2.2: icmp_req=2 ttl=64 time=0.025 ms
--- 10.2.2.2 ping statistics --2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.025/0.053/0.081/0.028 ms
student@vm1:~$ sudo ip link set dev tap0 down
student@vm1:~$ ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:d4:45:68 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
inet6 fe80::a00:27ff:fed4:4568/64 scope link
valid_lft forever preferred_lft forever
12: tap0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 500
link/ether ee:d8:2e:f6:bc:f1 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.1/24 brd 10.1.1.255 scope global tap0
inet 10.2.2.2/24 brd 10.2.2.255 scope global tap0
student@vm1:~$ ip r s
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
default via 10.0.2.2 dev eth0
student@vm1:~$ ping 10.2.2.2 -c 2
PING 10.2.2.2 (10.2.2.2) 56(84) bytes of data.
64 bytes from 10.2.2.2: icmp_req=1 ttl=64 time=0.037 ms
64 bytes from 10.2.2.2: icmp_req=2 ttl=64 time=0.024 ms
--- 10.2.2.2 ping statistics --2 packets transmitted, 2 received, 0% packet loss, time 999ms
96
rtt min/avg/max/mdev = 0.024/0.030/0.037/0.008 ms
student@vm1:~$ sudo tunctl -d tap0
Set 'tap0' nonpersistent
student@vm1:~$ ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:d4:45:68 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
inet6 fe80::a00:27ff:fed4:4568/64 scope link
valid_lft forever preferred_lft forever
student@vm1:~$ ls -al /sys/devices/virtual/net/tap0
ls: cannot access /sys/devices/virtual/net/tap0: No such file or directory
student@vm1:~$
Explanation
Install package for working with pseudo-(virtual)-interfaces.
Create pseudo-interface tap0.
Prints out contents of virtual directory created for this interfaces, containing its
settings and statistics.
Add to tap0 an IP address 10.1.1.1/24.
Print out current interface status.
Print out current routing table entries. Notice that Linux automatically added new
route for tap0.
Tests tap0 by sending ICMP echo request packet to it.
Brings tap0 to DOWN state.
Tests tap0 by sending ICMP echo request packet to it again. There will be an Extra
credit for you to explain why this still works despite intercase being down?
Brings tap0 to UP state.
Adds additional IP address 10.2.2.2/24 to tap0. ip a a is abbreviated version of ip
addr add. What this little + means you will find out on your own in Extra credit.
Prints out current interface status. Notice how ifconfig is unable to list new IP
address added using ip utility. Why? More Extra credit.
Print out current routing table. Notice that Linux automatically added one more route
for tap0.
Prints out current interface status using ip utility. You are able to see newly added
address here.
Prints our routing tables using ip utility.
Tests net tap0 IP address by sending ICMP echo request packet to it.
Brings tap0 to DOWN state.
Prints out current interface status.
Print out current routing table entries. Notice that tap0 routes are remove
automatically.
Tests net tap0 IP address by sending ICMP echo request packet to it. This works.
Why?
Removes pseudo-interface tap0.
Prints out current interface status. tap0 is absent.
Shows us that tap0 virtual directory is now gone as well.
Extra Credit
97
Familiarize yourself with man ifconfig, man ip, man netstat, man ss.
Why ping worked when tap0 was is in down state?
What brd + means?
Why ifconfig is unable to list new address you added using ip?
Exercise 25. Networking: configuration
files, /etc/network/interfaces
Configuring network interfaces from command line is all well, but now it is time to
learn how to tell vm1 to configure them automatically. For this you will learn about
the /etc/network/interfaces configuration file:
student@vm1:~$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
#(1) (2)
auto lo
#(3) (4)(5) (6)
iface lo inet loopback
# The primary network interface
#(7)
(8)
allow-hotplug eth0
#(9) (10) (11) (12)
iface eth0 inet dhcp
As usual, fields and their descriptions:
Field Description
(1) Automatically configure interface.
(2) Interface name.
(3) Start of interface configuration.
(4) Name of interface being configured.
(5) This interface uses TCP/IP networking, IPv4.
It is loopback interface. Default loopback address will be assigned to it
(6)
automatically.
Configure interface automatically when it becomes available (think usb-modem
(7)
here).
(8) Interface name.
(9) Start of interface configuration.
(10) Name of interface being configured.
(11) This interface uses TCP/IP networking, IPv4.
(12) This interface gets its parameters automatically via DHCP.
Other important files which contain network configuration, but we will not touch them
for here:
/etc/hosts — a computer file used in an operating system to map hostnames to IP
addresses. The hosts file is a plain text file and is conventionally named hosts.
98
/etc/hostname — a label that is assigned to a device connected to a computer
network and that is used to identify the device in various forms of electronic
communication.
/etc/resolv.conf — a computer file used in various operating systems to configure
the Domain Name System (DNS) resolver library. The file is a plain-text file usually
created by the network administrator or by applications that manage the
configuration tasks of the system. The resolvconf program is one such program on
linux machines which manages the resolv.conf file.
Let us remember tap0 from previous exercise. If you reboot vm1 it is gone. Of
course you can enable it by retyping relevant commands, but let us imagine that you
need it to become available automatically after reboot.
Now you will learn how to configure an interface using /etc/network/interfaces file.
Do this
1: ip a s
2: sudo vim /etc/network/interfaces
Now add this lines to the end of configuration file:
3: auto tap0
4: iface tap0 inet static
5:
address 10.2.2.2
6:
netmask 255.255.255.0
7:
tunctl_user uml-net
8:
9: allow-hotplug tap1
10: iface tap1 inet static
11:
address 10.3.3.3
12:
netmask 255.255.255.0
Now type :wq<ENTER> and continue:
13: sudo /etc/init.d/networking start
14: ip a s
15: sudo tunctl -t tap1 -u uml-net
16: ip a s
What you should see
student@vm1:~$ ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:d4:45:68 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
inet6 fe80::a00:27ff:fed4:4568/64 scope link
valid_lft forever preferred_lft forever
student@vm1:~$ sudo vim /etc/network/interfaces
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
allow-hotplug eth0
iface eth0 inet dhcp
auto tap0
99
iface tap0 inet static
address 10.2.2.2
netmask 255.255.255.0
tunctl_user uml-net
allow-hotplug tap1
iface tap1 inet static
address 10.3.3.3
netmask 255.255.255.0
~
"/etc/network/interfaces" 21L, 457C written
21,1-8
Bot
student@vm1:~$ sudo /etc/init.d/networking start
Configuring network interfaces...Set 'tap0' persistent and owned by uid 104 done.
student@vm1:~$ ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:d4:45:68 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
inet6 fe80::a00:27ff:fed4:4568/64 scope link
valid_lft forever preferred_lft forever
3: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN
qlen 500
link/ether 46:63:30:70:b5:21 brd ff:ff:ff:ff:ff:ff
inet 10.2.2.2/24 brd 10.2.2.255 scope global tap0
inet6 fe80::4463:30ff:fe70:b521/64 scope link
valid_lft forever preferred_lft forever
student@vm1:~$ sudo tunctl -t tap1 -u uml-net
Set 'tap1' persistent and owned by uid 104
student@vm1:~$ ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:d4:45:68 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
inet6 fe80::a00:27ff:fed4:4568/64 scope link
valid_lft forever preferred_lft forever
3: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN
qlen 500
link/ether 46:63:30:70:b5:21 brd ff:ff:ff:ff:ff:ff
inet 10.2.2.2/24 brd 10.2.2.255 scope global tap0
inet6 fe80::4463:30ff:fe70:b521/64 scope link
valid_lft forever preferred_lft forever
4: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN
qlen 500
link/ether 8a:ed:90:33:93:55 brd ff:ff:ff:ff:ff:ff
inet 10.3.3.3/24 brd 10.3.3.255 scope global tap1
inet6 fe80::88ed:90ff:fe33:9355/64 scope link
valid_lft forever preferred_lft forever
student@vm1:~$
Explanation
Print out current interface configuration.
Edit /etc/network/interfaces.
100
Automatically configure tap0.
Set the following IPv4 static parameters fop tap0.
Add IP address 10.2.2.2 to tap0.
Specify netmask for this IP address. Parameters broadcast and network are derived
from this netmask automatically.
Specify user who will own tap0 interface.
Empty line for readability.
Add the following parameters to tap1 interface when it appears in the system.
Set the following IPv4 static parameters fop tap1.
Add IP address 10.3.3.3 to tap0.
Specify netmask for this IP address.
Applies changes in network configuration.
Prints out current interface configuration. You are able to see that tap0 is add to the
list of interfaces.
Adds tap1 pseudo-interface.
Prints out current interface configuration. You are able to see that its parameters
specified in /etc/network/interfaces are applied to it automatically.
Extra credit
Explain how network and broadcast parameters are derived.
Try this: ping kitty. Failure is expected. Now add an entry to /etc/hosts so that you
will be able to successfully ping.
Exercise 26. Networking: packet filter
configuration, iptables
Let me start with quoting Wikipedia on iptables:
Iptables is a user space application program that allows a system administrator to
configure the tables provided by the Linux kernel firewall (implemented as different
Netfilter modules) and the chains and rules it stores. Different kernel modules and
programs are currently used for different protocols; iptables applies to IPv4, ip6tables
to IPv6, arptables to ARP, and ebtables to Ethernet frames.
To use it you must understand the following concepts:
LINKTYPE_LINUX_SLL — tcpdump pseudo link layer protocol.
Ethernet frame header — A data packet on an Ethernet link is called an Ethernet
frame. A frame begins with preamble and start frame delimiter. Following which,
each Ethernet frame continues with an Ethernet header featuring destination and
source MAC addresses. The middle section of the frame is payload data including
any headers for other protocols (e.g. Internet Protocol) carried in the frame. The
frame ends with a 32-bit cyclic redundancy check which is used to detect any
corruption of data in transit.
IPv4 header — An IP packet consists of a header section and a data section. The
IPv4 packet header consists of 14 fields, of which 13 are required. The 14th field is
optional and aptly named: options.
TCP segment structure — Transmission Control Protocol accepts data from a data
stream, segments it into chunks, and adds a TCP header creating a TCP segment.
The TCP segment is then encapsulated into an Internet Protocol (IP) datagram. A
101
TCP segment is “the packet of information that TCP uses to exchange data with its
peers.”
I remind you about guides to get them:
Read corresponding Wikipedia articles until you achieve at least superficial
understanding (but good old hardcore grinding is of course better).
Watch this videos from http://www.visualland.net/ (another link
www.visualland.net.cn):
Expand IP Address tree node in the left of the site and work you way through it.
Expand TCP tree node and do the same.
Read Linux networking concepts introduction. This guide is good because it even
acknowledges that The Internet is for porn.
I am not able to describe iptables any better than Peter Harrison's excellent guide. If
you never used it, you should check this guide first.
But what I will, however, put theory into practice and show what happens inside of
iptalbes step-by-step in a very simple scenario of data exchange. First things first,
main concepts:
iptables — program which is used to set up, maintain, and inspect the tables of IPv4
packet filter rules in the Linux kernel. Several different tables may be defined. Each
table contains a number of built-in chains and may also contain user-defined chains.
ip6tables — same for IPv6.
chain — list of rules which can match a set of packets. Each rule specifies what to
do with a packet that matches. This is called a target, which may be a jump to a
user-defined chain in the same table.
target — A firewall rule specifies criteria for a packet and a target. If the packet does
not match, the next rule in the chain is the examined; if it does match, then the next
rule is specified by the value of the target, which can be the name of a user-defined
chain or one of the special values:
ACCEPT — let the packet through.
DROP — drop the packet on the floor.
QUEUE — pass the packet to userspace.
RETURN — stop traversing this chain and resume at the next rule in the previous
(calling) chain. If the end of a built-in chain is reached or a rule in a built-in chain with
target RETURN is matched, the target specified by the chain policy determines the
fate of the packet.
Now let us see what default tables and built-in chains there are:
Table
Built-in chains Description
name
This table is used mainly for configuring exemptions from
connection tracking in combination with the NOTRACK
target. It registers at the netfilter hooks with higher priority
and is thus called before ip_conntrack, or any other IP
raw
tables.
PREROUTING For packets arriving via any network interface.
OUTPUT
For packets generated by local processes.
This table is used for specialized packet alteration.
PREROUTING For altering incoming packets before routing.
mangle
OUTPUT
For altering locally-generated packets before routing.
INPUT
For packets coming into the box itself.
102
FORWARD
For altering packets being routed through the box.
POSTROUTING For altering packets as they are about to go out.
This table is consulted when a packet that creates a new
connection is encountered.
PREROUTING For altering packets as soon as they come in.
nat
OUTPUT
For altering locally-generated packets before routing.
POSTROUTING For altering packets as they are about to go out.
This is the default table (if no -t option is passed).
INPUT
For packets destined to local sockets.
filter
FORWARD
For packets being routed through the box.
OUTPUT
For locally-generated packets.
Okay, we are ready to see how this works in real life. I will send a string Hello world!
to vm1 from my home computer using TCP protocol and netcat utility which is just
like cat, but over the network. The setup:
1. I forwarded another port, 80, to my home PC running Linux so am able to connect
like this:
(My home PC) --> vm1:80
2. I added this rules to iptables to log all what happens with network packets inside
iptables:
sudo iptables -t raw -A PREROUTING -p tcp -m tcp --dport 80 -j TRACE
sudo iptables -t raw -A INPUT -p tcp -m tcp --sport 80 -j TRACE
This is my exact iptables ruleset on vm1:
root@vm1:/home/student# for i in raw mangle nat filter ; do echo -e "\n-----" TABLE: $i '-----' ; iptables t $i -L ; done
----- TABLE: raw ----Chain PREROUTING (policy ACCEPT) target prot opt source
destination
TRACE
tcp -- anywhere
anywhere
tcp dpt:www
Chain OUTPUT (policy ACCEPT)
target prot opt source
destination
TRACE
tcp -- anywhere
anywhere
tcp spt:www
----- TABLE: mangle ----Chain PREROUTING (policy ACCEPT) target prot opt source
destination
Chain INPUT (policy ACCEPT)
target prot opt source
destination
Chain FORWARD (policy ACCEPT) target prot opt source
destination
Chain OUTPUT (policy ACCEPT)
target prot opt source
destination
Chain POSTROUTING (policy ACCEPT) target prot opt source
destination
----- TABLE: nat ----Chain PREROUTING (policy ACCEPT) target prot opt source
Chain POSTROUTING (policy ACCEPT) target prot opt source
Chain OUTPUT (policy ACCEPT)
target prot opt source
----- TABLE: filter ----Chain INPUT (policy ACCEPT)
target prot opt source
Chain FORWARD (policy ACCEPT) target prot opt source
Chain OUTPUT (policy ACCEPT)
target prot opt source
destination
destination
destination
destination
destination
destination
As you can see, there are no other rules. Another way to look at iptables rules is to
use iptables-save utility:
root@vm1:/home/student# iptables-save
# Generated by iptables-save v1.4.8 on Fri Jul 13 08:09:04 2012
#(1)
*mangle
#(2)
(3) (4) (5)
103
:PREROUTING ACCEPT [15662:802240]
:INPUT ACCEPT [15662:802240]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [12756:3671974]
:POSTROUTING ACCEPT [12756:3671974]
COMMIT
# Completed on Fri Jul 13 08:09:04 2012
# Generated by iptables-save v1.4.8 on Fri Jul 13 08:09:04 2012
*nat
:PREROUTING ACCEPT [18:792]
:POSTROUTING ACCEPT [42:2660]
:OUTPUT ACCEPT [42:2660]
COMMIT
# Completed on Fri Jul 13 08:09:04 2012
# Generated by iptables-save v1.4.8 on Fri Jul 13 08:09:04 2012
*raw
:PREROUTING ACCEPT [15854:814892]
:OUTPUT ACCEPT [12855:3682054]
-A PREROUTING -p tcp -m tcp --dport 80 -j TRACE
-A OUTPUT -p tcp -m tcp --sport 80 -j TRACE
COMMIT
# Completed on Fri Jul 13 08:09:04 2012
# Generated by iptables-save v1.4.8 on Fri Jul 13 08:09:04 2012
*filter
:INPUT ACCEPT [35107:2459066]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [26433:10670628]
COMMIT
# Completed on Fri Jul 13 08:09:04 2012
The iptables-save fiels are as follows:
Field Description
(1) Table name
(2) Chain name
(3) Chain policy
(4) Packet counter
(5) Byte counter
3. I set up nc to listen on port 80:
nc -l 80
4. I sent a string to vm1 using nc:
echo 'Hello, world!' | nc localhost 80
The following exchange between my home pc and vm1 happaned:
08:00:05.655339 IP 10.0.2.2.51534 > 10.0.2.15.80: Flags [S], seq 4164179969, win 65535, options
[mss 1460], length 0
08:00:05.655653 IP 10.0.2.15.80 > 10.0.2.2.51534: Flags [S.], seq 4149908960, ack 4164179970,
win 5840, options [mss 1460], length 0
08:00:05.655773 IP 10.0.2.2.51534 > 10.0.2.15.80: Flags [.], ack 1, win 65535, length 0
08:00:05.655868 IP 10.0.2.2.51534 > 10.0.2.15.80: Flags [P.], seq 1:15, ack 1, win 65535, length 14
08:00:05.655978 IP 10.0.2.15.80 > 10.0.2.2.51534: Flags [.], ack 15, win 5840, length 0
08:00:10.037978 IP 10.0.2.2.51534 > 10.0.2.15.80: Flags [F.], seq 15, ack 1, win 65535, length 0
08:00:10.038287 IP 10.0.2.15.80 > 10.0.2.2.51534: Flags [F.], seq 1, ack 16, win 5840, length 0
08:00:10.038993 IP 10.0.2.2.51534 > 10.0.2.15.80: Flags [.], ack 2, win 65535, length 0
Let us remember how data is transfered. To do this, let us take apart first packet:
#
(13) (15) (14)
(16)
(20) (17)
(25)
08:00:05.655339 IP 10.0.2.2.51534 > 10.0.2.15.80: Flags [S], seq 4164179969, win 65535,
#
(8)
(9)
options [mss 1460], length 0
104
#
#
#
#
#
#
#
#
(1) (2) (3) (4)
(5)
____ ____ ____ ___________________ ____
0x0000: 0000 0001 0006 5254 0012 3502 0000 0800 ......RT..5.....
(6) (7) (8) (9)(10,11)(12) (13)
____ ____ ____ ____ /\/\ ____ _________
0x0010: 4500 002c a006 0000 4006 c2b5 0a00 0202 E..,....@.......
(14)
(15) (16) (17)
(18)
_________ ____ ____ _________ __________
0x0020: 0a00 020f c94e 0050 f834 5801 0000 0000 .....N.P.4X.....
(19,20)(21) (22) (23) (24) (25)
/\/\ ____ ____ ____ ____ ____
0x0030: 6002 ffff 6641 0000 0204 05b4 0000
`...fA........
`
Field and descriptions in our packet:
DOD
OSI model
model
Field in
layer
layer
Link
Physical/Data LINUX_SLL
link
header
Field Description
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
Internet
Network
IPv4 header
(9)
(10)
(11)
(12)
(13)
(14)
(15)
(16)
(17)
(18)
Transport
Transport
TCP header
(19)
(20)
(21)
(22)
(23)
(24)
105
Packet type.
ARPHRD_ type.
Link-layer (MAC) address length.
Linkk-layer (MAC) source address.
Protocol type (IP).
Version, Internet Header Length,
Differentiated Services Code Point,
Explicit Congestion Notification.
Total Length.
Identification, is primarily used for
uniquely identifying fragments of an
original IP datagram.
Flags, Fragment offset.
Time to live (TTL).
Protocol number.
Header checksum.
Source IP address.
Destination IP address.
Source TCP port.
Destination TCP port.
TCP initial sequence number.
ACK number field (empty because it
is first packet).
SYN TCP flag.
TCP window size.
TCP checksum.
Urgent pointer.
Options field start
TCP maximum segment size
(maximum transfer unit - 40 bytes).
And now let us see what happens to this packet in iptables:
(25)
#(1)(2)
(3) (4) (5) (6) (7) (8)
(9)
(10) (11)
(12)
raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
#
(13) (14) (15)
(16) (17) (18)
(19)
(20)
LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=40966 PROTO=TCP SPT=51534
DPT=80
#
(21)
(22) (23)
(24) (25)(26) (27)
SEQ=4164179969 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT
(020405B4)
mangle:PREROUTING:policy:1 IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=40966 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179969 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT
(020405B4)
nat:PREROUTING:policy:1 IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=40966 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179969 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT
(020405B4)
mangle:INPUT:policy:1
IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=40966 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179969 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT
(020405B4)
filter:INPUT:policy:1
IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=40966 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179969 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT
(020405B4)
Fields and descrptions of iptables log:
Field Description
(1) Table name.
(2) Chain name.
(3) Type (policy for the policy of the built in chains.
(4) Rule number.
(5) Intput interface.
(6) Output interface (empty, because packet is destinied for vm1 itself)
(7) MAC addresses
(8) Destination (vm1) MAC
(9) Source MAC
(10) Protocol type IP.
(11) Source IP address
(12) Destination IP address
(13) IP packet length in bytes (excluding Link Layer header)
(14) IP type of Service.
106
(15) IP precedence.
(16) IP time to live.
(17) IP packet ID.
(18) Protocol type TCP.
(19) TCP source port.
(20) TCP destination port.
(21) TCP sequence number.
(22) TCP acknoledgement number.
(23) TCP window size.
(24) TCP reserved bits.
(25) TCP SYN flag is set.
(25) TCP urgent pointer is not set.
(25) TCP options.
Now I will show you this exchange side-by-side (more like paragraph by paragraph)
using tcpdump output and iptables log. You task will be to go through this exchange
line by line and understand what happens. I recommend you to print this exchange
and work through it using pen and paper, you can print it from special page. The
questions you need to answer are this:
What does each field mean? Take a pencil and connect field from tcpdump trace to
raw packet data in hex to iptables log.
In what order packets are processed? Which table first, which last, why?
Why only first packet is processed through nat table?
This is the output, behold:
08:00:05.655339 IP 10.0.2.2.51534 > 10.0.2.15.80: Flags [S], seq 4164179969, win 65535, options
[mss 1460], length 0
0x0000: 0000 0001 0006 5254 0012 3502 0000 0800 ......RT..5.....
0x0010: 4500 002c a006 0000 4006 c2b5 0a00 0202 E..,....@.......
0x0020: 0a00 020f c94e 0050 f834 5801 0000 0000 .....N.P.4X.....
0x0030: 6002 ffff 6641 0000 0204 05b4 0000
`...fA........
`
raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=40966 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179969 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT
(020405B4)
mangle:PREROUTING:policy:1 IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=40966 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179969 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT
(020405B4)
nat:PREROUTING:policy:1 IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=40966 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179969 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT
(020405B4)
mangle:INPUT:policy:1
IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=40966 PROTO=TCP SPT=51534
DPT=80
107
SEQ=4164179969 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT
(020405B4)
filter:INPUT:policy:1
IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=40966 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179969 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT
(020405B4)
08:00:05.655653 IP 10.0.2.15.80 > 10.0.2.2.51534: Flags [S.], seq 4149908960, ack 4164179970,
win 5840, options [mss 1460], length 0
0x0000: 0004 0001 0006 0800 27d4 4568 0000 0800 ........'.Eh....
0x0010: 4500 002c 0000 4000 4006 22bc 0a00 020f E..,..@.@.".....
0x0020: 0a00 0202 0050 c94e f75a 95e0 f834 5802 .....P.N.Z...4X.
0x0030: 6012 16d0 c224 0000 0204 05b4
`....$......
'
raw:OUTPUT:policy:2
IN= OUT=eth0 SRC=10.0.2.15 DST=10.0.2.2
LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80
DPT=51534
SEQ=4149908960 ACK=4164179970 WINDOW=5840 RES=0x00 ACK SYN
URGP=0 OPT (020405B4) UID=0 GID=0
mangle:OUTPUT:policy:1
IN= OUT=eth0 SRC=10.0.2.15 DST=10.0.2.2
LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80
DPT=51534
SEQ=4149908960 ACK=4164179970 WINDOW=5840 RES=0x00 ACK SYN
URGP=0 OPT (020405B4) UID=0 GID=0
filter:OUTPUT:policy:1
IN= OUT=eth0 SRC=10.0.2.15 DST=10.0.2.2
LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80
DPT=51534
SEQ=4149908960 ACK=4164179970 WINDOW=5840 RES=0x00 ACK SYN
URGP=0 OPT (020405B4) UID=0 GID=0
mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=10.0.2.15 DST=10.0.2.2
LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80
DPT=51534
SEQ=4149908960 ACK=4164179970 WINDOW=5840 RES=0x00 ACK SYN
URGP=0 OPT (020405B4) UID=0 GID=0
08:00:05.655773 IP 10.0.2.2.51534 > 10.0.2.15.80: Flags [.], ack 1, win 65535, length 0
0x0000: 0000 0001 0006 5254 0012 3502 0000 0800 ......RT..5.....
0x0010: 4500 0028 a007 0000 4006 c2b8 0a00 0202 E..(....@.......
0x0020: 0a00 020f c94e 0050 f834 5802 f75a 95e1 .....N.P.4X..Z..
0x0030: 5010 ffff f0b1 0000 0000 0000 0000
P.............
raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40967 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179970 ACK=4149908961 WINDOW=65535 RES=0x00 ACK URGP=0
mangle:PREROUTING:policy:1 IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40967 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179970 ACK=4149908961 WINDOW=65535 RES=0x00 ACK URGP=0
mangle:INPUT:policy:1
IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40967 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179970 ACK=4149908961 WINDOW=65535 RES=0x00 ACK URGP=0
filter:INPUT:policy:1
IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
108
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40967 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179970 ACK=4149908961 WINDOW=65535 RES=0x00 ACK URGP=0
08:00:05.655868 IP 10.0.2.2.51534 > 10.0.2.15.80: Flags [P.], seq 1:15, ack 1, win 65535, length 14
0x0000: 0000 0001 0006 5254 0012 3502 0000 0800 ......RT..5.....
0x0010: 4500 0036 a008 0000 4006 c2a9 0a00 0202 E..6....@.......
0x0020: 0a00 020f c94e 0050 f834 5802 f75a 95e1 .....N.P.4X..Z..
0x0030: 5018 ffff af45 0000 4865 6c6c 6f2c 2077 P....E..Hello,.w
0x0040: 6f72 6c64 210a
orld!.
raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=54 TOS=0x00 PREC=0x00 TTL=64 ID=40968 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179970 ACK=4149908961 WINDOW=65535 RES=0x00 ACK PSH
URGP=0
mangle:PREROUTING:policy:1 IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=54 TOS=0x00 PREC=0x00 TTL=64 ID=40968 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179970 ACK=4149908961 WINDOW=65535 RES=0x00 ACK PSH
URGP=0
mangle:INPUT:policy:1
IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=54 TOS=0x00 PREC=0x00 TTL=64 ID=40968 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179970 ACK=4149908961 WINDOW=65535 RES=0x00 ACK PSH
URGP=0
filter:INPUT:policy:1
IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=54 TOS=0x00 PREC=0x00 TTL=64 ID=40968 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179970 ACK=4149908961 WINDOW=65535 RES=0x00 ACK PSH
URGP=0
08:00:05.655978 IP 10.0.2.15.80 > 10.0.2.2.51534: Flags [.], ack 15, win 5840, length 0
0x0000: 0004 0001 0006 0800 27d4 4568 0000 0800 ........'.Eh....
0x0010: 4500 0028 377c 4000 4006 eb43 0a00 020f E..(7|@.@..C....
0x0020: 0a00 0202 0050 c94e f75a 95e1 f834 5810 .....P.N.Z...4X.
0x0030: 5010 16d0 d9d3 0000
P.......
'
raw:OUTPUT:policy:2
IN= OUT=eth0 SRC=10.0.2.15 DST=10.0.2.2
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=14204 DF PROTO=TCP SPT=80
DPT=51534
SEQ=4149908961 ACK=4164179984 WINDOW=5840 RES=0x00 ACK URGP=0
mangle:OUTPUT:policy:1
IN= OUT=eth0 SRC=10.0.2.15 DST=10.0.2.2
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=14204 DF PROTO=TCP SPT=80
DPT=51534
SEQ=4149908961 ACK=4164179984 WINDOW=5840 RES=0x00 ACK URGP=0
filter:OUTPUT:policy:1
IN= OUT=eth0 SRC=10.0.2.15 DST=10.0.2.2
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=14204 DF PROTO=TCP SPT=80
DPT=51534
SEQ=4149908961 ACK=4164179984 WINDOW=5840 RES=0x00 ACK URGP=0
mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=10.0.2.15 DST=10.0.2.2
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=14204 DF PROTO=TCP SPT=80
DPT=51534
SEQ=4149908961 ACK=4164179984 WINDOW=5840 RES=0x00 ACK URGP=0
08:00:10.037978 IP 10.0.2.2.51534 > 10.0.2.15.80: Flags [F.], seq 15, ack 1, win 65535, length 0
109
0x0000:
0x0010:
0x0020:
0x0030:
0000 0001 0006 5254 0012 3502 0000 0800 ......RT..5.....
4500 0028 a00e 0000 4006 c2b1 0a00 0202 E..(....@.......
0a00 020f c94e 0050 f834 5810 f75a 95e1 .....N.P.4X..Z..
5011 ffff f0a2 0000 0000 0000 0000
P.............
raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40974 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179984 ACK=4149908961 WINDOW=65535 RES=0x00 ACK FIN
URGP=0
mangle:PREROUTING:policy:1 IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40974 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179984 ACK=4149908961 WINDOW=65535 RES=0x00 ACK FIN
URGP=0
mangle:INPUT:policy:1
IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40974 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179984 ACK=4149908961 WINDOW=65535 RES=0x00 ACK FIN
URGP=0
filter:INPUT:policy:1
IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40974 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179984 ACK=4149908961 WINDOW=65535 RES=0x00 ACK FIN
URGP=0
08:00:10.038287 IP 10.0.2.15.80 > 10.0.2.2.51534: Flags [F.], seq 1, ack 16, win 5840, length 0
0x0000: 0004 0001 0006 0800 27d4 4568 0000 0800 ........'.Eh....
0x0010: 4500 0028 377d 4000 4006 eb42 0a00 020f E..(7}@.@..B....
0x0020: 0a00 0202 0050 c94e f75a 95e1 f834 5811 .....P.N.Z...4X.
0x0030: 5011 16d0 d9d1 0000
P.......
'
raw:OUTPUT:policy:2
IN= OUT=eth0 SRC=10.0.2.15 DST=10.0.2.2
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=14205 DF PROTO=TCP SPT=80
DPT=51534
SEQ=4149908961 ACK=4164179985 WINDOW=5840 RES=0x00 ACK FIN
URGP=0 UID=0 GID=0
mangle:OUTPUT:policy:1
IN= OUT=eth0 SRC=10.0.2.15 DST=10.0.2.2
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=14205 DF PROTO=TCP SPT=80
DPT=51534
SEQ=4149908961 ACK=4164179985 WINDOW=5840 RES=0x00 ACK FIN
URGP=0 UID=0 GID=0
filter:OUTPUT:policy:1
IN= OUT=eth0 SRC=10.0.2.15 DST=10.0.2.2
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=14205 DF PROTO=TCP SPT=80
DPT=51534
SEQ=4149908961 ACK=4164179985 WINDOW=5840 RES=0x00 ACK FIN
URGP=0 UID=0 GID=0
mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=10.0.2.15 DST=10.0.2.2
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=14205 DF PROTO=TCP SPT=80
DPT=51534
SEQ=4149908961 ACK=4164179985 WINDOW=5840 RES=0x00 ACK FIN
URGP=0 UID=0 GID=0
08:00:10.038993 IP 10.0.2.2.51534 > 10.0.2.15.80: Flags [.], ack 2, win 65535, length 0
0x0000: 0000 0001 0006 5254 0012 3502 0000 0800 ......RT..5.....
0x0010: 4500 0028 a00f 0000 4006 c2b0 0a00 0202 E..(....@.......
110
0x0020: 0a00 020f c94e 0050 f834 5811 f75a 95e2 .....N.P.4X..Z..
0x0030: 5010 ffff f0a1 0000 0000 0000 0000
P.............
raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40975 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179985 ACK=4149908962 WINDOW=65535 RES=0x00 ACK URGP=0
mangle:PREROUTING:policy:1 IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40975 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179985 ACK=4149908962 WINDOW=65535 RES=0x00 ACK URGP=0
mangle:INPUT:policy:1
IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40975 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179985 ACK=4149908962 WINDOW=65535 RES=0x00 ACK URGP=0
filter:INPUT:policy:1
IN=eth0 OUT= MAC=08:00:27:d4:45:68:52:54:00:12:35:02:08:00
SRC=10.0.2.2 DST=10.0.2.15
LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=40975 PROTO=TCP SPT=51534
DPT=80
SEQ=4164179985 ACK=4149908962 WINDOW=65535 RES=0x00 ACK URGP=0
Now you will learn how to set up basic iptables rules and enable iptables logging.
Do this
1: sudo iptables-save
2: sudo iptables -t filter -A INPUT -i lo -j ACCEPT
3: sudo iptables -t filter -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
4: sudo iptables -t filter -P INPUT DROP
5: sudo iptables -nt filter -L --line-numbers
6: ping -c 2 -W 1 10.0.2.2
7: sudo iptables -t filter -A INPUT --match state --state ESTABLISHED -j ACCEPT
8: sudo iptables -nt filter -L --line-numbers
9: ping -c 2 -W 1 10.0.2.2
10: sudo modprobe ipt_LOG
11: sudo iptables -nt raw -L --line-numbers
12: sudo iptables -t raw -A PREROUTING -p udp -m udp --dport 1024 -j TRACE
13: sudo iptables -t raw -A OUTPUT -p udp -m udp --sport 1024 -j TRACE
14: sudo tail -n0 -f /var/log/kern.log | cut -c52-300 &
15: nc -ulp 1024 &
16: echo 'Hello there!' | nc -u localhost 1000
17: <CTRL+C>
18: fg
19: <CTRL+C>
20: fg
21: <CTRL+C>
What you should see
student@vm1:~$ sudo iptables-save
# Generated by iptables-save v1.4.8 on Mon Jul 16 09:01:32 2012
*mangle
:PREROUTING ACCEPT [45783:3411367]
:INPUT ACCEPT [45783:3411367]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [30409:9552110]
:POSTROUTING ACCEPT [30331:9543294]
111
COMMIT
# Completed on Mon Jul 16 09:01:32 2012
# Generated by iptables-save v1.4.8 on Mon Jul 16 09:01:32 2012
*nat
:PREROUTING ACCEPT [24:1056]
:POSTROUTING ACCEPT [755:41247]
:OUTPUT ACCEPT [817:45207]
COMMIT
# Completed on Mon Jul 16 09:01:32 2012
# Generated by iptables-save v1.4.8 on Mon Jul 16 09:01:32 2012
*raw
:PREROUTING ACCEPT [3171:197900]
:OUTPUT ACCEPT [1991:1294054]
-A PREROUTING -p udp -m udp --dport 80 -j TRACE
-A OUTPUT -p udp -m udp --sport 80 -j TRACE
COMMIT
# Completed on Mon Jul 16 09:01:32 2012
# Generated by iptables-save v1.4.8 on Mon Jul 16 09:01:32 2012
*filter
:INPUT ACCEPT [54:3564]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [28:2540]
COMMIT
# Completed on Mon Jul 16 09:01:32 2012
student@vm1:~$ sudo iptables -t filter -A INPUT -i lo -j ACCEPT
student@vm1:~$ sudo iptables -t filter -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
student@vm1:~$ sudo iptables -t filter -P INPUT DROP
student@vm1:~$ sudo iptables -nt filter -L --line-numbers
Chain INPUT (policy DROP)
num target prot opt source
destination
1 ACCEPT all -- 0.0.0.0/0
0.0.0.0/0
2 ACCEPT tcp -- 0.0.0.0/0
0.0.0.0/0
tcp dpt:22
Chain FORWARD (policy ACCEPT)
num target prot opt source
destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source
destination
student@vm1:~$ ping -c 2 -W 1 10.0.2.2
PING 10.0.2.2 (10.0.2.2) 56(84) bytes of data.
--- 10.0.2.2 ping statistics --2 packets transmitted, 0 received, 100% packet loss, time 1008ms
student@vm1:~$ sudo iptables -t filter -A INPUT --match state --state ESTABLISHED -j ACCEPT
student@vm1:~$ sudo iptables -nt filter -L --line-numbers
Chain INPUT (policy DROP)
num target prot opt source
destination
1 ACCEPT all -- 0.0.0.0/0
0.0.0.0/0
2 ACCEPT tcp -- 0.0.0.0/0
0.0.0.0/0
tcp dpt:22
3 ACCEPT all -- 0.0.0.0/0
0.0.0.0/0
state ESTABLISHED
Chain FORWARD (policy ACCEPT)
num target prot opt source
destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source
destination
student@vm1:~$ ping -c 2 -W 1 10.0.2.2
112
PING 10.0.2.2 (10.0.2.2) 56(84) bytes of data.
64 bytes from 10.0.2.2: icmp_req=1 ttl=63 time=0.385 ms
64 bytes from 10.0.2.2: icmp_req=2 ttl=63 time=0.142 ms
--- 10.0.2.2 ping statistics --2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.142/0.263/0.385/0.122 ms
student@vm1:~$ sudo iptables -nt raw -L --line-numbers
Chain PREROUTING (policy ACCEPT)
num target prot opt source
destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source
destination
student@vm1:~$ sudo iptables -t raw -A PREROUTING -p udp -m udp --dport 1024 -j TRACE
student@vm1:~$ sudo iptables -t raw -A OUTPUT -p udp -m udp --sport 1024 -j TRACE
student@vm1:~$ sudo iptables -nt raw -L --line-numbers
Chain PREROUTING (policy ACCEPT)
num target prot opt source
destination
1 TRACE
udp -- 0.0.0.0/0
0.0.0.0/0
udp dpt:1024
Chain OUTPUT (policy ACCEPT)
num target prot opt source
destination
1 TRACE
udp -- 0.0.0.0/0
0.0.0.0/0
udp spt:1024
student@vm1:~$ sudo tail -n0 -f /var/log/kern.log | cut -c52-300 &
[1] 10249
student@vm1:~$ nc -ulp 1024 &
[2] 10251
student@vm1:~$ echo 'Hello there!' | nc -u localhost 1024
Hello there!
raw:PREROUTING:policy:2 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00
SRC=127.0.0.1 DST=127.0.0.1 LEN=41 TOS=0x00 PREC=0x00 TTL=64 ID=57898 DF
PROTO=UDP SPT=50407 DPT=1024 LEN=21
mangle:PREROUTING:policy:1 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00
SRC=127.0.0.1 DST=127.0.0.1 LEN=41 TOS=0x00 PREC=0x00 TTL=64 ID=57898 DF
PROTO=UDP SPT=50407 DPT=1024 LEN=21
mangle:INPUT:policy:1 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00
SRC=127.0.0.1 DST=127.0.0.1 LEN=41 TOS=0x00 PREC=0x00 TTL=64 ID=57898 DF
PROTO=UDP SPT=50407 DPT=1024 LEN=21
filter:INPUT:rule:1 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1
DST=127.0.0.1 LEN=41 TOS=0x00 PREC=0x00 TTL=64 ID=57898 DF PROTO=UDP SPT=50407
DPT=1024 LEN=21
^C
[2]+ Stopped
nc -ulp 1024
student@vm1:~$ fg
nc -ulp 1024
^C
student@vm1:~$ fg
sudo tail -n0 -f /var/log/kern.log | cut -c52-300
^C
student@vm1:~$
Explanation
Prints out all iptables rules from all tables. You can see that there are none.
Allows all incoming traffic on lo (loopback) interface.
Allows all incoming traffic to TCP port 22, which is ssh.
Changes default INPUT policy to DROP, forbidding all incoming connections except
explicitly allowed to TCP port 22. If you loss connection to vm1 here, it means you
did something wrong, reboot it in VirtualBox and try again.
113
Lists current filter rules. Note: you can delete rules by number, like this: sudo
iptables -t filter -D INPUT 2. Notice how policy is totally not the same as rule.
Tries to ping your default gateway, failing to do this. Why it is so if outgoing
connections are allowed (Chain OUTPUT (policy ACCEPT))? Well, outgoing
packets are being sent to gateway, but reply from gateway is not able to get in.
Adds a rule telling iptables to allow all packets which belong to already established
connections, for example to all connections originating from vm1.
Lists current filter rules. You are able to see our new rule.
Pings vm1 default gateway, this time successfully.
Loads Linux kernel module which allows to use packet filter logging facility.
Adds rule to log all incoming packets which are destined to UDP port 1024 of any
interface of vm1.
Adds rule to log all outgoing packets which originate from UDP port 1024 of any
interface of vm1.
Lists raw table rules.
Starts tail in the background which will print all new lines written to
/var/log/kern.log. cut will remove unnecessary prefix of log entries in the beginning.
Notice how background process is able to write to your terminal.
Starts nc in server mode, listening on UDP port 1024 on all interfaces of vm1.
Starts nc in client mode, sending string Hello there! to our server mode nc. tail
prints out all new lines in kern.log and you are able to see how our single UDP
packet goes from table to table inside of Linux kernel packet filter. There is no reply,
so only one single packet is being sent and processed.
Kills client mode nc.
Brings server mode nc to foreground.
Kills server mode nc.
Bring sudo tail -n0 -f /var/log/kern.log | cut -c52-300 & to foreground.
Kills it.
Extra credit
This was big by itself. Just print out this log and work through it using pencil until you
really understand what is going on in every field of every line. If you will stuck, ask
away here. http://nixsrv.com/llthw/ex26/log
Exercise 27. Networking: secure shell,
ssh, sshd, scp
You may already know that SSH is a network protocol which allows you to login into
vm1 over the network. Let us look into in in some detail.
Secure Shell (SSH) is a network protocol for secure data communication, remote
shell services or command execution and other secure network services between
two networked computers that it connects via a secure channel over an insecure
network: a server and a client (running SSH server and SSH client programs,
respectively). The protocol specification distinguishes two major versions that are
referred to as SSH-1 and SSH-2.
The best-known application of the protocol is for access to shell accounts on Unixlike operating systems. It was designed as a replacement for Telnet and other
insecure remote shell protocols such as the Berkeley rsh and rexec protocols, which
114
send information, notably passwords, in plaintext, rendering them susceptible to
interception and disclosure using packet analysis. The encryption used by SSH is
intended to provide confidentiality and integrity of data over an unsecured network,
such as the Internet.
The important SSH programs, concepts and configuration files:
OpenSSH — open source ssh programs implementation.
ssh — the client program which allows you to connect to ssh server. Putty is such
client program for example.
sshd — the server program which allows you to connect to it with ssh.
/etc/ssh/ssh_config — default client program configuration file.
/etc/ssh/sshd_config — default server program configuration file.
Public-key cryptography — a cryptographic system requiring two separate keys,
one of which is secret and one of which is public. Although different, the two parts of
the key pair are mathematically linked. One key locks or encrypts the plaintext, and
the other unlocks or decrypts the cyphertext. Neither key can perform both functions.
One of these keys is published or public and the other is kept private.
SSH keys — SSH uses public-key cryptography to authenticate the remote
computer and allow it to authenticate the user, if necessary. Anyone can produce a
matching pair of different key—s (public and private). The public key is placed on all
computers that must allow access to the owner of the matching private key (the
owner keeps the private key secret). While authentication is based on the private
key, the key itself is never transferred through the network during authentication.
/etc/ssh/moduli — prime numbers and generators for use by sshd(8) in the DiffieHellman Group Exchange key exchange method.
/etc/ssh/ssh_host_dsa_key, /etc/ssh/ssh_host_rsa_key — private host RSA and
DSA keys.
/etc/ssh/ssh_host_dsa_key.pub, /etc/ssh/ssh_host_rsa_key.pub — public host
RSA and DSA keys.
SSH protocol is so important, so widely used and has so many capabilities, that
some understanding of its workings is mandatory. This is some of its uses:
scp — file transfer over ssh.
sftp — ftp-like protocol for managing remote files.
sshfs — remote filesystem mounting over ssh.
ssh tunnelling — a method to transfer almost any data over secure connection.
This is so important because it can be used as a foundation for construction of
protected systems, among many its other uses.
To get some understanding of this protocol let us see what happens during ssh
session. To this we will start with examining the annotated output of connecting from
vm1 to vm1 (yes, it can be done, and it is perfectly valid to do). The outline:
YOU
Type ssh vm1
Control is now passed to SSH client
SSH client
Enters plaintext phase
Reads configuration
Negotiats protocol with SSH server
Enters SSH-transport phase
Negotiates with SSH server
Data encryption chiphers
Data integrity algorithms
Data compression algorithms
Start key exchange using Diffie-Hellman algorithm
115
Resulting shared key is used to establish secured connection
Enters SSH-userauth phase
Asks you to enter password
Control is now passed to you
YOU
Enter you password
Control is now passed to SSH client
SSH client
Authenticates you on SSH server
Enters SSH-connection phase
Allocates pseudo teminal for YOU
Starts a shell for YOU
Control is now passed to YOU
YOU
Do something u(n)useful on vm1
Close shell
Control is now passed to ssh client
SSH client
Closes pseudo terminal
Closes connection
Now read this:
The OpenSSH Protocol under the Hood.
http://www.cs.ust.hk/faculty/cding/COMP581/SLIDES/slide24.pdf SSH - Secure
SHell presentation.
And examine the real output of ssh session:
student@vm1:~$ ssh -vv vm1
Protocol version selection, plaintext
------------------------------------OpenSSH_5.5p1 Debian-6+squeeze2, OpenSSL 0.9.8o 01 Jun 2010
# Speaks for itself, I will mark such entries with -- below
debug1: Reading configuration data /etc/ssh/ssh_config
# Applying default options for all hosts. Additional options for each host may be
# specified in the configuration file
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to vm1 [127.0.1.1] port 22.
debug1: Connection established.
debug1: identity file /home/student/.ssh/id_rsa type -1
# no such files
debug1: identity file /home/student/.ssh/id_rsa-cert type -1
debug1: identity file /home/student/.ssh/id_dsa type -1
debug1: identity file /home/student/.ssh/id_dsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.5p1 Debian-6+squeeze2
debug1: match: OpenSSH_5.5p1 Debian-6+squeeze2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.5p1 Debian-6+squeeze2
debug2: fd 3 setting O_NONBLOCK
SSH-transport, binary packet protocol
------------------------------------debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
# Key exchange algorithms
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchangesha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
# SSH host key types
116
debug2: kex_parse_kexinit: ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,sshrsa,ssh-dss
# Data encryption ciphers
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
# Data integrity algorithms
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmacripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmacripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
# Data compression algorithms
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows
debug2: kex_parse_kexinit: reserved 0
# Messages back from server
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchangesha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmacripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmacripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
# Message authentication code setup
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
# Key exchange
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 135/256
debug2: bits set: 498/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
# Server authentication. vm1 host key is not known because it is our first connection
debug2: no key of type 0 for host vm1
debug2: no key of type 2 for host vm1
# Confirmation of host key acceptance
The authenticity of host 'vm1 '(127.0.1.1)' can't be established.
RSA key fingerprint is b6:06:92:5e:04:49:d9:e8:57:90:61:1b:16:87:bb:09.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'vm1' (RSA) to the list of known hosts.
# Key is added to /home/student/.ssh/known_hosts and checked
debug2: bits set: 499/1024
117
debug1: ssh_rsa_verify: signature correct
# Based on shared master key, data encryption key and data integrity key are derived
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
# Information about this is sent to server
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
# IP roaming not enabled? Not sure about this.
debug1: Roaming not allowed by server
SSH-userauth
-----------debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/student/.ssh/id_rsa ((nil))
debug2: key: /home/student/.ssh/id_dsa ((nil))
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/student/.ssh/id_rsa
debug1: Trying private key: /home/student/.ssh/id_dsa
debug2: we did not send a packet, disable method
debug1: Next authentication method: password
student@vm1''s password:
debug2: we sent a password packet, wait for reply
debug1: Authentication succeeded (password).
SSH-connection
-------------debug1: channel 0: new [client-session]
debug2: channel 0: send open
# Disable SSH mutiplexing.
# More info: http://www.linuxjournal.com/content/speed-multiple-ssh-connections-same-server
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
# Sending environment variables
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug2: channel 0: request shell confirm 1
# Set TCP_NODELAY flag: http://en.wikipedia.org/wiki/Nagle%27s_algorithm
debug2: fd 3 setting TCP_NODELAY
debug2: callback done
# Connection opened
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
# Pseudo terminal allocation
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
# Shell is started
debug2: shell request accepted on channel 0
# Loggin in is completed
118
Linux vm1 2.6.32-5-amd64 #1 SMP Sun May 6 04:00:17 UTC 2012 x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
You have mail.
Last login: Thu Jul 19 05:14:40 2012 from 10.0.2.2
student@vm1:~$ debug2: client_check_window_change: changed
debug2: channel 0: request window-change confirm 0
student@vm1:~$ debug2: client_check_window_change: changed
debug2: channel 0: request window-change confirm 0
student@vm1:~$ logout
Ending ssh connection
--------------------debug2: channel 0: rcvd eof # end of file
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
# signalling that channels are half-closed for writing, through a channel protocol extension
# notification "eow@openssh.com" http://www.openssh.com/txt/release-5.1
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug2: channel 0: rcvd eow
# Ending connection
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug2: channel 0: rcvd close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
Connection to vm1 closed.
Transferred: sent 1928, received 2632 bytes, in 93.2 seconds
Bytes per second: sent 20.7, received 28.2
debug1: Exit status 0
student@vm1:~$
Now you will learn how to start sshd in debug mode, set up public key authentication
and copy files using scp.
Do this
1: mkdir -v ssh_test
2: cd ssh_test
3: cp -v /etc/ssh/sshd_config .
4: sed -i'.bak' 's/^Port 22$/Port 1024/' sshd_config
5: sed -i 's/^HostKey \/etc\/ssh\/ssh_host_rsa_key$/Hostkey
\/home\/student\/ssh_test\/ssh_host_rsa_key/' sshd_config
6: sed -i 's/^HostKey \/etc\/ssh\/ssh_host_dsa_key$/Hostkey
\/home\/student\/ssh_test\/ssh_host_dsa_key/' sshd_config
7: diff sshd_config.bak sshd_config
8: ssh-keygen -b 4096 -t rsa -N '' -v -h -f ssh_host_rsa_key
9: ssh-keygen -b 1024 -t dsa -N '' -v -h -f ssh_host_dsa_key
119
10: ssh-keygen -b 4096 -t rsa -N '' -v -f ~/.ssh/id_rsa
11: cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
12: /usr/sbin/sshd -Ddf sshd_config > sshd.out 2>&1 &
13: ssh-keyscan -H vm1 127.0.0.1 >> ~/.ssh/known_hosts
14: /usr/sbin/sshd -Ddf sshd_config >> sshd.out 2>&1 &
15: ssh vm1 -v -p 1024 2>ssh.out
16: ps au --forest
17: logout
18: /usr/sbin/sshd -Ddf sshd_config >> sshd.out 2>&1 &
19: scp -v -P 1024 vm1:.bashrc . 2>scp.out
What you should see
student@vm1:~$ mkdir -v ssh_test
mkdir: created directory 'ssh_test'
student@vm1:~$ cd ssh_test
student@vm1:~/ssh_test$ cp -v /etc/ssh/sshd_config .
'/etc/ssh/sshd_config' -> './sshd_config'
student@vm1:~/ssh_test$ sed -i'.bak' 's/^Port 22$/Port 1024/' sshd_config
student@vm1:~/ssh_test$ sed -i 's/^HostKey \/etc\/ssh\/ssh_host_rsa_key$/Hostkey
\/home\/student\/ssh_test\/ssh_host_rsa_key/' sshd_config
student@vm1:~/ssh_test$ sed -i 's/^HostKey \/etc\/ssh\/ssh_host_dsa_key$/Hostkey
\/home\/student\/ssh_test\/ssh_host_dsa_key/' sshd_config
student@vm1:~/ssh_test$ diff sshd_config.bak sshd_config
5c5
< Port 22
--> Port 1024
11,12c11,12
< HostKey /etc/ssh/ssh_host_rsa_key
< HostKey /etc/ssh/ssh_host_dsa_key
--> Hostkey /home/student/ssh_test/ssh_host_rsa_key
> Hostkey /home/student/ssh_test/ssh_host_dsa_key
student@vm1:~/ssh_test$ ssh-keygen -b 4096 -t rsa -N '' -v -h -f ssh_host_rsa_key
Generating public/private rsa key pair.
Your identification has been saved in ssh_host_rsa_key.
Your public key has been saved in ssh_host_rsa_key.pub.
The key fingerprint is:
8c:0a:8d:ae:c7:34:e6:29:9c:c2:14:29:b8:d9:1d:34 student@vm1
'The key's randomart image is:
+--[ RSA 4096]----+
|
|
| E
|
|. .. .
|
|oo o. o
|
|.++.... S
|
|oo=...
|
|+=oo.
|
|o==
|
|oo
|
+-----------------+
student@vm1:~/ssh_test$ ssh-keygen -b 1024 -t dsa -N '' -v -h -f ssh_host_dsa_key
Generating public/private dsa key pair.
Your identification has been saved in ssh_host_dsa_key.
Your public key has been saved in ssh_host_dsa_key.pub.
The key fingerprint is:
cd:6b:2a:a2:ba:80:65:71:85:ef:2e:6a:c0:a7:d9:aa student@vm1
'The key's randomart image is:
120
+--[ DSA 1024]----+
| ..
|
| ..
|
| . ..
|
| o . o
|
|. o . S o
|
|o+ . . . |
|o.= . o
|
|.o..o o o
|
|E=+o o ..
|
+-----------------+
student@vm1:~/ssh_test$ ssh-keygen -b 4096 -t rsa -N '' -v -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /home/student/.ssh/id_rsa.
Your public key has been saved in /home/student/.ssh/id_rsa.pub.
The key fingerprint is:
50:65:18:61:3f:41:36:07:4f:40:36:a7:4b:6d:64:28 student@vm1
'The key's randomart image is:
+--[ RSA 4096]----+
|
=B&+* |
|
oE=.& |
|
. .= + |
|
..+ |
|
S.
|
|
|
|
|
|
|
|
|
+-----------------+
student@vm1:~/ssh_test$ cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
student@vm1:~/ssh_test$ /usr/sbin/sshd -Ddf sshd_config > sshd.out 2>&1 &
[2] 26896
student@vm1:~/ssh_test$ ssh-keyscan -H vm1 127.0.0.1 >> ~/.ssh/known_hosts
# 127.0.0.1 SSH-2.0-OpenSSH_5.5p1 Debian-6+squeeze2
# vm1 SSH-2.0-OpenSSH_5.5p1 Debian-6+squeeze2
[2]+ Exit 255
/usr/sbin/sshd -Ddf sshd_config > sshd.out 2>&1
student@vm1:~/ssh_test$ /usr/sbin/sshd -Ddf sshd_config >> sshd.out 2>&1 &
[1] 26957
student@vm1:~/ssh_test$ ssh vm1 -v -p 1024 2>ssh.out
Linux vm1 2.6.32-5-amd64 #1 SMP Sun May 6 04:00:17 UTC 2012 x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
You have mail.
Last login: Fri Jul 20 09:10:30 2012 from vm1.site
Environment:
LANG=en_US.UTF-8
USER=student
LOGNAME=student
HOME=/home/student
PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
MAIL=/var/mail/student
SHELL=/bin/bash
SSH_CLIENT=127.0.1.1 47456 1024
SSH_CONNECTION=127.0.1.1 47456 127.0.1.1 1024
SSH_TTY=/dev/pts/0
121
TERM=xterm
student@vm1:~$ ps au --forest
USER
PID %CPU %MEM VSZ RSS TTY
STAT START TIME COMMAND
student 26224 0.0 1.2 23660 6576 pts/2 Ss 09:09 0:01 -bash
student 27020 1.0 0.6 68392 3236 pts/2 S 09:50 0:00 \_ sshd: student [priv]
student 27025 0.0 0.2 68392 1412 pts/2 S 09:50 0:00 | \_ sshd: student@pts/0
student 27026 9.0 1.2 23564 6404 pts/0 Ss 09:50 0:00 |
\_ -bash
student 27051 0.0 0.2 16308 1060 pts/0 R+ 09:50 0:00 |
\_ ps au --forest
student 27021 1.1 0.5 38504 2880 pts/2 S+ 09:50 0:00 \_ ssh vm1 -v -p 1024
root
1107 0.0 0.1 5932 620 tty6 Ss+ Jul18 0:00 /sbin/getty 38400 tty6
root
1106 0.0 0.1 5932 616 tty5 Ss+ Jul18 0:00 /sbin/getty 38400 tty5
root
1105 0.0 0.1 5932 620 tty4 Ss+ Jul18 0:00 /sbin/getty 38400 tty4
root
1104 0.0 0.1 5932 620 tty3 Ss+ Jul18 0:00 /sbin/getty 38400 tty3
root
1103 0.0 0.1 5932 616 tty2 Ss+ Jul18 0:00 /sbin/getty 38400 tty2
root
1102 0.0 0.1 5932 616 tty1 Ss+ Jul18 0:00 /sbin/getty 38400 tty1
student@vm1:~$ logout
student@vm1:~/ssh_test$
[1]+ Exit 255
/usr/sbin/sshd -Ddf sshd_config > sshd.out 2>&1
student@vm1:~/ssh_test$ /usr/sbin/sshd -Ddf sshd_config >> sshd.out 2>&1 &
[1] 27067
student@vm1:~/ssh_test$ scp -v -P 1024 vm1:.bashrc . 2>scp.out
Environment:
LANG=en_US.UTF-8
USER=student
LOGNAME=student
HOME=/home/student
PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
MAIL=/var/mail/student
SHELL=/bin/bash
SSH_CLIENT=127.0.1.1 47459 1024
SSH_CONNECTION=127.0.1.1 47459 127.0.1.1 1024
.bashrc
100% 3184
3.1KB/s 00:00
[1]+ Exit 255
/usr/sbin/sshd -Ddf sshd_config >> sshd.out 2>&1
Explanation
Creates /home/student/ssh_test directory.
Make it current working directory.
Copies sshd_config to this directory.
Changes sshd listen port from 22 to 1024, making copy named sshd_config.bak in
the process.
Replaces RSA host key localtion.
Replaces DSA host key localtion.
Shows difference between old and new versions of sshd_config.
Generates new 4096-bit RSA host key pair with empty pass-phrase, saving it to
/home/student/ssh_test/ssh_host_rsa_key and
/home/student/ssh_test/ssh_host_rsa_key.pub.
The same, but for DSA key.
Generates new authentication key pair, saving it to /home/student/.ssh/id_rsa and
/home/student/.ssh/id_rsa.pub.
Copies id_rsa.pub to /home/student/.ssh/authorized_keys to allow passwordless
authentication.
Starts new SSH server on port 1024 in debug mode, saving all output to sshd.log.
Extracts host authentication key for SSH client and puts it to
/home/student/.ssh/known_hosts.
122
Starts new SSH server on port 1024 in debug mode, appending all output to
sshd.log. This is because in debug mode SSH server servers only only one
connection.
Connects to this server using ssh client.
Prints out current running processes in tree format. You are able to see that you are
working in shell bash started by sshd serving your connection which in turn is
started by sshd you started yourself several lines before.
Logs out from ssh session.
Starts SSH server once more.
Copies file .bashrc from your home directory to your current directory.
Extra credit
Watch this videos explaining how encryption works:
http://www.youtube.com/watch?v=3QnD2c4Xovk
Read this: http://docstore.mik.ua/orelly/networking_2ndEd/ssh/ch03_04.htm
Read debug output found in files ssh.out, scp.out and sshd.out. Explain to yourself
what is going on.
Exercise 28. Performance: getting
performance stats, uptime, free, top
This exercise is pretty simple. First, what kind of performance stats do we need?
CPU usage stats:
How much is it loaded?
Which processes are using it?
Memory usage stats:
How much memory is used?
How much memory is free?
How much memory is used for caching?
Which processes consume it?
Disk usage stats:
How many input/output operations are performed?
By which processes?
Network usage stats:
How many data is transferred?
By which processes?
Process stats:
How many processes there are?
What they are doing? Working, waiting for something?
If waiting for something what is it? CPU, disks, network?
To get this stats we can use the following utilities:
uptime — how long the system has been running.
free — display amount of free and used memory in the system.
vmstat — information about processes, memory, paging, block IO, traps, disks and
cpu activity.
top — display Linux tasks in real-time.
Let us examine this programs and their output.
uptime output:
123
student@vm1:~$ uptime
#(1)
(2)
(3)
(4) (5) (6)
03:13:58 up 4 days, 22:45, 1 user, load average: 0.00, 0.00, 0.00
Fields and descriptions:
Field Description
(1) Current time.
(2) Uptime (time passed since boot).
(3) How many users are currently logged in.
CPU load in past 1 minute. This is not normalized, so a load average of 1
(4) means a single CPU system is loaded all the time while on a 4 CPU system it
means it was idle 75% of the time.
(5) CPU load in past 5 minutes.
(6) CPU load in past 15 minutes.
free output:
student@vm1:~$ free -mt
#
(1)
(2)
(3)
(4)
(5)
(6)
total
used
free shared buffers cached
Mem:
496
267
229
0
27
196
#
(7)
(8)
-/+ buffers/cache:
43
453
#9
Swap:
461
0
461
# 10
Total:
958
267
691
Fields and descriptions:
Field Description
(1) Total amount of psysical memory.
(2) Total amount of used physical memory.
(3) Total mount of free physical memory.
(4) Shared memory column should be ignored; it is obsolete.
(5) Total RAM dedicated to cache disk block, and filesystem metadata.
(6) Total RAM dedicated to pages from file reading.
(7) Total amount of physical memory excluding buffers and cache, (2) - (5) - (6)
Total amount of free physical memeory counting buffers and cache as free, (1)
(8)
- (7)
(9) Swap file usage statistics.
(10) Total memory usage statistics, including swap
vmstat output:
student@vm1:~$ vmstat -S M
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---#(1,2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12,13,14,15,16)
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0
0 229 27 196 0 0 0 0 11 6 0 0 100 0
student@vm1:~$ vmstat -S M -a
#
(17) (18)
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---r b swpd free inact active si so bi bo in cs us sy id wa
0 0
0 11 434 19 0 0 24 2 11 6 0 0 100 0
124
student@vm1:~$ vmstat -d
#19 (20) (21) (22) (23) (24) (25) (26) (27) (28) (29)
disk- ------------reads------------ ------------writes----------- -----IO-----total merged sectors
ms total merged sectors
ms cur sec
sda 11706 353 402980 17612 9303 40546 336358 46980
0
sr0
0
0
0
0
0
0
0
0
0
0
loop0
0
0
0
0
0
0
0
0
0
0
19
student@vm1:~$ vmstat -m | head
#(30)
(31) (32) (33) (34)
Cache
Num Total Size Pages
ext3_inode_cache
13700 13700 808 10
ext3_xattr
0
0 88 46
journal_handle
170 170 24 170
journal_head
37 72 112 36
revoke_table
256 256 16 256
revoke_record
128 128 32 128
kmalloc_dma-512
8
8 512
8
ip6_dst_cache
16 24 320 12
UDPLITEv6
0
0 1024
8
Fields and descriptions:
Mode
Stats Field Description
(1) r: The number of processes waiting for run time.
Procs
(2) b: The number of processes in uninterruptible sleep.
(3) swpd: the amount of virtual memory used.
(4) free: the amount of idle memory.
(5) buff: the amount of memory used as buffers.
Memory
(6) cache: the amount of memory used as cache.
(17) inact: the amount of inactive memory.
(18) active: the amount of active memory.
(7) si: Amount of memory swapped in from disk (/s).
Swap
(8) so: Amount of memory swapped to disk (/s).
Virtual
(9) bi: Blocks received from a block device (blocks/s).
I/O
memory
(10) bo: Blocks sent to a block device (blocks/s).
in: The number of interrupts per second, including the
(11)
clock.
System
(12) cs: The number of context switches per second.
us: Time spent running non-kernel code. (user time,
(13)
including nice time)
(14) sy: Time spent running kernel code. (system time)
CPU
id: Time spent idle. Prior to Linux 2.5.41, this includes
(15)
IO-wait time.
wa: Time spent waiting for IO. Prior to Linux 2.5.41,
(16)
included in idle.
Device (19) Device name
(20) total: Total reads completed successfully
Disk, -d
Reads (21) merged: grouped reads (resulting in one I/O)
(22) sectors: Sectors read successfully
125
Writes
I/O
Slab, -m
Slab
(23)
(24)
(25)
(26)
(27)
(28)
(29)
(30)
(31)
(32)
(33)
(34)
ms: milliseconds spent reading
total: Total writes completed successfully
merged: grouped writes (resulting in one I/O)
sectors: Sectors written successfully
ms: milliseconds spent writing
cur: I/O in progress
s: seconds spent for I/O
cache: Cache name
num: Number of currently active objects
total: Total number of available objects
size: Size of each object
pages: Number of pages with at least one active object
top output:
# (1)
(2)
(3)
(4)
top - 03:22:44 up 4 days, 22:54, 1 user, load average: 0.00, 0.00, 0.00
#
(5)
(6)
(7)
(8)
(9)
Tasks: 63 total, 1 running, 62 sleeping, 0 stopped, 0 zombie
#
(10) (11) (12) (13)
(14) (15) (16) (17)
Cpu(s): 0.0%us, 1.1%sy, 0.0%ni, 98.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
#
(18)
(19)
(20)
(21)
Mem: 508820k total, 273792k used, 235028k free, 27844k buffers
#
(22)
(23)
(24)
(25)
Swap: 473080k total,
0k used, 473080k free, 201252k cached
#(26) (27) (28)(29) (30) (31) (32,33) (34)(35)
(36) (37)
PID USER
PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root
20 0 8356 804 676 S 0.0 0.2 0:05.99 init
2 root
20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root
RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
4 root
20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0
5 root
RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
6 root
20 0 0 0 0 S 0.0 0.0 0:03.25 events/0
7 root
20 0 0 0 0 S 0.0 0.0 0:00.00 cpuset
<...>
Fields and descriptions:
Section Field
Description
(1)
Current time.
(2)
Uptime (time passed since boot).
(3)
How many users are currently logged in.
Uptime
CPU load in past 1, 5 and 15 minuts. This is not normalized,
so a load average of 1 means a single CPU system is loaded
(4)
all the time while on a 4 CPU system it means it was idle 75%
of the time.
(5)
Total number of running processes.
(6)
Number of processes currently executing.
(7)
Number of processes currently sleeping.
Tasks
Number of processes curretnly stopped (with CTRL+Z for
(8)
example).
(9)
Number of defunct (“zombie”) process, terminated but not
126
reaped by its parent.
The time the CPU has spent running users' processes that are
(10)
not niced.
The time the CPU has spent running the kernel and its
(11)
processes.
The time the CPU has spent running users' proccess that
(12)
have been niced.
(13)
The time the CPU spent idle.
Cpu(s)
(14)
Amount of time the CPU has been waiting for I/O to complete.
The amount of time the CPU has been servicing hardware
(15)
interrupts.
The amount of time the CPU has been servicing software
(16)
interrupts.
The amount of CPU 'stolen' from this virtual machine by the
(17)
hypervisor for other tasks (such as running another virtual
machine).
(18)
Total amount of psysical memory.
(19)
Total amount of used physical memory.
(20)
Total mount of free physical memory.
Mem/swap
Total RAM dedicated to cache disk block, and filesystem
(21)
metadata.
(22,23,24) Total, used and free swap memory.
(25)
Total RAM dedicated to pages from file reading.
The task's unique process ID, which periodically wraps,
(26)
though never restarting at zero.
(27)
The effective user name of the task's owner.
(28)
The priority of the task.
The nice value of the task. A negative nice value means
higher priority, whereas a positive nice value means lower
(29)
priority. Zero in this field simply means priority will not be
adjusted in determining a task's dispatchability.
The total amount of virtual memory used by the task. It
includes all code, data and shared libraries plus pages that
(30)
have been swapped out and pages that have been mapped
Process
but not used.
(31)
The non-swapped physical memory a task has used.
The amount of shared memory used by a task. It simply
(32)
reflects memory that could be potentially shared with other
processes.
The status of the task which can be one of: 'D' =
(33)
uninterruptible sleep, 'R' = running, 'S' = sleeping, 'T' = traced
or stopped, 'Z' = zombie.
The task's share of the elapsed CPU time since the last
(34)
screen update, expressed as a percentage of total CPU time.
(35)
A task's currently used share of available physical memory.
127
CPU Time, hundredths, The same as 'TIME', but reflecting
more granularity through hundredths of a second.
(37)
Command – Command line or Program name
There are lots of fields as you may see. Many of fields are present in several utilities,
which duplicate functionality of each other somewhat. Normally you will need only
small subset of this fields, but you need to know that there is this many (actually, a
lot more) information about system performance available to you, because
sometimes an obscure problem happens and to be able to solve it you need to know
how to read this stats.
Now you will learn how to uses system performance utilities.
(36)
Do this
1: uptime
2: free
3: vmstat
4: ( sleep 5 && dd if=/dev/urandom of=/dev/null bs=1M count=30 && sleep 5 && killall vmstat )&
vmstat 1
5: uptime
6: ( sleep 5 && dd if=/dev/zero of=test.img bs=32 count=$((32*1024*200)) && sleep 5 && killall
vmstat )& vmstat -nd 1 | egrep -v 'loop|sr0'
7: echo 3 | sudo tee /proc/sys/vm/drop_caches
8: free -mt ; find / >/dev/null 2>&1 ; free -mt
9: echo 3 | sudo tee /proc/sys/vm/drop_caches
10: cat test.img /dev/null ; free -mt
What you should see
student@vm1:~$ uptime
05:36:45 up 6 days, 1:08, 1 user, load average: 0.00, 0.00, 0.00
student@vm1:~$ free
total
used
free shared buffers cached
Mem:
508820 239992 268828
0
820 213720
-/+ buffers/cache:
25452 483368
Swap:
473080
0 473080
student@vm1:~$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---r b swpd free buff cache si so bi bo in cs us sy id wa
0 0
0 268828 820 213720 0 0 21 10 14 11 0 0 100 0
student@vm1:~$ ( sleep 5 && dd if=/dev/urandom of=/dev/null bs=1M count=30 && sleep 5 && killall
vmstat )& vmstat 1
[1] 6078
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---r b swpd free buff cache si so bi bo in cs us sy id wa
1 1
0 268556 828 213736 0 0 21 10 14 11 0 0 100 0
0 0
0 268556 828 213772 0 0 16 0 19 10 0 0 100 0
0 0
0 268556 828 213772 0 0 0 0 13 8 0 0 100 0
0 0
0 268556 828 213772 0 0 0 0 15 11 0 0 100 0
0 0
0 268556 828 213772 0 0 0 0 14 10 0 0 100 0
0 0
0 268556 828 213772 0 0 0 0 18 13 0 0 100 0
1 0
0 267316 836 213844 0 0 74 0 267 26 0 99 1 0
1 0
0 267316 836 213844 0 0 0 0 303 7 0 100 0 0
1 0
0 267316 836 213844 0 0 0 0 271 11 0 100 0 0
1 0
0 267316 836 213844 0 0 0 0 257 12 0 100 0 0
30+0 records in
30+0 records out
31457280 bytes (31 MB) copied, 4.95038 s, 6.4 MB/s
0 0
0 267928 860 213860 0 0 27 0 265 29 1 97 2 0
0 0
0 267936 860 213848 0 0 0 0 15 9 0 0 100 0
128
0 0
0 267936 860 213848 0 0 0 0 14 7 0 0 100 0
0 0
0 267936 860 213848 0 0 0 0 14 7 0 0 100 0
0 0
0 267936 860 213848 0 0 0 0 13 11 0 0 100 0
Terminated
student@vm1:~$ uptime
05:22:15 up 6 days, 54 min, 1 user, load average: 0.07, 0.02, 0.00
[1]+ Done
( sleep 5 && dd if=/dev/urandom of=/dev/null bs=1M count=30 && sleep 5 &&
killall vmstat )
student@vm1:~$ uptime
05:22:22 up 6 days, 54 min, 1 user, load average: 0.06, 0.02, 0.00
student@vm1:~$ ( sleep 5 && dd if=/dev/zero of=test.img bs=32 count=$((32*1024*200)) && sleep 5
&& killall vmstat )& vmstat -nd 1 | egrep -v 'loop|sr0'
[1] 6086
disk- ------------reads------------ ------------writes----------- -----IO-----total merged sectors
ms total merged sectors
ms cur sec
sda 146985 2230744 21821320 105848 32190 1343154 10927338 1330144
0 105
sda 146995 2230744 21821648 105848 32190 1343154 10927338 1330144
0 105
sda 146995 2230744 21821648 105848 32190 1343154 10927338 1330144
0 105
sda 146995 2230744 21821648 105848 32190 1343154 10927338 1330144
0 105
sda 146995 2230744 21821648 105848 32190 1343154 10927338 1330144
0 105
sda 146995 2230744 21821648 105848 32190 1343154 10927338 1330144
0 105
sda 146999 2230744 21821680 105856 32190 1343154 10927338 1330144
0 105
sda 146999 2230744 21821680 105856 32190 1343154 10927338 1330144
0 105
sda 147000 2230744 21821688 105856 32208 1344160 10935530 1330288
0 105
sda 147000 2230744 21821688 105856 32274 1349214 10976490 1330748
0 105
sda 147000 2230744 21821688 105856 32325 1353259 11009258 1331236
0 105
sda 147000 2230744 21821688 105856 32450 1364657 11101442 1337176
0 105
sda 147000 2230744 21821688 105856 32450 1364657 11101442 1337176
0 105
sda 147001 2230744 21821696 105856 32471 1366301 11114762 1337348
0 105
sda 147001 2230744 21821696 105856 32525 1370529 11149018 1337732
0 105
sda 147001 2230744 21821696 105856 32573 1374577 11181786 1338064
0 105
sda 147001 2230744 21821696 105856 32698 1386562 11278666 1346244
0 105
6553600+0 records in
6553600+0 records out
209715200 bytes (210 MB) copied, 11.7088 s, 17.9 MB/s
sda 147001 2230744 21821696 105856 32698 1386562 11278666 1346244
0 105
sda 147001 2230744 21821696 105856 32698 1386562 11278666 1346244
0 105
sda 147001 2230744 21821696 105856 32698 1386562 11278666 1346244
0 105
sda 147001 2230744 21821696 105856 32698 1386562 11278666 1346244
0 105
sda 147001 2230744 21821696 105856 32762 1393910 11337962 1349192
0 105
student@vm1:~$ echo 3 | sudo tee /proc/sys/vm/drop_caches
3
[1]+ Done
( sleep 5 && dd if=/dev/zero of=test.img bs=32 count=$((32*1024*200)) &&
sleep 5 && killall vmstat )
student@vm1:~$ free -mt ; find / >/dev/null 2>&1 ; free -mt
total
used
free shared buffers cached
Mem:
496
30
466
0
0
5
-/+ buffers/cache:
24
472
Swap:
461
0
461
Total:
958
30
928
total
used
free shared buffers cached
Mem:
496
64
432
0
22
6
-/+ buffers/cache:
35
461
Swap:
461
0
461
Total:
958
64
894
student@vm1:~$ echo 3 | sudo tee /proc/sys/vm/drop_caches
3
student@vm1:~$ cat test.img /dev/null ; free -mt
total
used
free shared buffers cached
Mem:
496
230
265
0
0
205
129
-/+ buffers/cache:
Swap:
461
Total:
958
student@vm1:~$
24
0
230
471
461
727
Explanation
Prints out current uptime.
Prints out free memory information.
This one is interesting. It is better be thought like a sort of experiment. First, we
launch ( sleep 5 && dd if=/dev/urandom of=/dev/null bs=1M count=30 && sleep
5 && killall vmstat )& in the background, and immediately after that we start vmstat
in continuous mode so it will print out its information until interrupted. We are able to
see that after 5 seconds since this command start CPU load increases dramatically
for some time and then decreases, after another 5 seconds vmstat is killed.
Prints out current uptime. Notice changes in load average.
This is another experiment, almost the same as before, but this time with disk writes.
Drops all caches and buffers.
Another experiment. We want to see how reading all file and directory names in the
system affects filesystem cache in memory, and we are able to see that it is cached
in buffers, which is in accordance to theory.
Drops all caches and buffers again.
This time we want to see how reading a file affects filesystem cache in memory. We
are able to see that files which are read are cached in cached section for increasing
time of consequent access.
Extra credit
Why in our first experiment not user, but system CPU usage goes up to 100?
What this means? dd if=/dev/zero of=test.img bs=32 count=$( (32*1024*200) ).
Launch top and press h. Now try sorting its output by CPU, memory and PID.
Exercise 29. Kernel: kernel messages,
dmesg
Well, if you made it this far it is time to talk about kernel. We will start this talk with
Wikipedia definition of an Operating System kernel:
In computing, the kernel (from German Kern, nucleus, core) is the main component
of most computer operating systems; it is a bridge between applications and the
actual data processing done at the hardware level. The kernel's responsibilities
include managing the system's resources (the communication between hardware
and software components). Usually as a basic component of an operating system, a
kernel can provide the lowest-level abstraction layer for the resources (especially
processors and I/O devices) that application software must control to perform its
function. It typically makes these facilities available to application processes through
inter-process communication mechanisms and system calls.
And this is what Wikipedia tells us about Linux Kernel specifically:
The Linux kernel is the operating system kernel used by the Linux family of Unix-like
operating systems. It is one of the most prominent examples of free and open source
software. It supports true preemptive multitasking (both in user mode and kernel
130
mode), virtual memory, shared libraries, demand loading, shared copy-on-write
executables, memory management, the Internet protocol suite, and threading.
Now is a good time to visit corresponding Wikipedia article and spend some time
clicking feverishly on all those scary terms which describe technical features Linux
Kernel has. After doing so, let us talk about much more mundane subject, which is a
way kernel cat tell us something. This happens, for example, if USB-stick is attached
to the computer, or network link becomes down, or filesystem mounting is
performed. To be able to tell you all this things, kernel uses a mechanism known as
display message or driver message, name of which is abbreviated to dmesg.
This mechanism is represented by the buffer of fixed size to which kernel writes its
messages. On Debian Linux distribution information from this buffer are also copied
to /var/log/dmseg after system log daemon is started. This is done to preserve this
messages which would otherwise be overwritten by new ones.
dmesg is also name of utility which allows you to view those messages which are
currently in kernel buffer, and also change this buffer size.
Let me summarize dmesg-related files and programs:
dmesg — print or control the kernel ring buffer
/var/log/dmseg — log file in Debian distribution which contains copy of dmesg
messages during system boot only, without timestamps.
/var/log/kern.log — log file in Debian distribution which contains copy of all dmesg
messages, including timestamps. Notice that this timestamps start ticking after
rsyslog logging daemon is started, which means that all boot time messages before
rsyslog is started will have the same timestamp. This file contains /var/log/dmesg
in itself.
/var/log/messages — log file in Debian distribution which logs all non-debug and
non-critical messages. It contains /var/log/dmesg in itself.
/var/log/syslog — log file in Debian distribution which logs everything, except auth
related messages. It contains all messages which go to /var/log/messages and
/var/log/kern.log in itself.
Do this
1: date
2: sudo umount /tmp ; sudo mount /tmp
3: sudo tail -f /var/log/dmesg /var/log/messages /var/log/syslog /var/log/kern.log
What you should see
student@vm1:~$ date
Tue Jul 24 06:55:33 EDT 2012
student@vm1:~$ sudo umount /tmp ; sudo mount /tmp
student@vm1:~$ dmesg | tail
[ 7.166240] tun: Universal TUN/TAP device driver, 1.6
[ 7.166242] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[ 7.432019] ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 7.435270] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 7.435927] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 17.472049] tap0: no IPv6 routers present
[ 17.592044] eth0: no IPv6 routers present
[ 217.497357] kjournald starting. Commit interval 5 seconds
[ 217.497561] EXT3 FS on sda8, internal journal
[ 217.497564] EXT3-fs: mounted filesystem with ordered data mode.
student@vm1:~$ sudo tail /var/log/dmesg /var/log/messages /var/log/syslog /var/log/kern.log
==> /var/log/dmesg <==
[ 6.762569] EXT3 FS on sda5, internal journal
[ 6.762572] EXT3-fs: mounted filesystem with ordered data mode.
131
[
[
[
[
[
[
[
[
6.767237] kjournald starting. Commit interval 5 seconds
6.767407] EXT3 FS on sda6, internal journal
6.767410] EXT3-fs: mounted filesystem with ordered data mode.
7.166240] tun: Universal TUN/TAP device driver, 1.6
7.166242] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
7.432019] ADDRCONF(NETDEV_UP): eth0: link is not ready
7.435270] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
7.435927] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
==> /var/log/messages <==
Jul 24 06:52:07 vm1 kernel: [ 6.767407] EXT3 FS on sda6, internal journal
Jul 24 06:52:07 vm1 kernel: [ 6.767410] EXT3-fs: mounted filesystem with ordered data mode.
Jul 24 06:52:07 vm1 kernel: [ 7.166240] tun: Universal TUN/TAP device driver, 1.6
Jul 24 06:52:07 vm1 kernel: [ 7.166242] tun: (C) 1999-2004 Max Krasnyansky
<maxk@qualcomm.com>
Jul 24 06:52:07 vm1 kernel: [ 7.432019] ADDRCONF(NETDEV_UP): eth0: link is not ready
Jul 24 06:52:07 vm1 kernel: [ 7.435270] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: RX
Jul 24 06:52:07 vm1 kernel: [ 7.435927] ADDRCONF(NETDEV_CHANGE): eth0: link becomes
ready
Jul 24 06:55:36 vm1 kernel: [ 217.497357] kjournald starting. Commit interval 5 seconds
Jul 24 06:55:36 vm1 kernel: [ 217.497561] EXT3 FS on sda8, internal journal
Jul 24 06:55:36 vm1 kernel: [ 217.497564] EXT3-fs: mounted filesystem with ordered data mode.
==> /var/log/syslog <==
Jul 24 06:52:08 vm1 acpid: 1 rule loaded
Jul 24 06:52:08 vm1 acpid: waiting for events: event logging is off
Jul 24 06:52:08 vm1 /usr/sbin/cron[882]: (CRON) INFO (pidfile fd = 3)
Jul 24 06:52:08 vm1 /usr/sbin/cron[883]: (CRON) STARTUP (fork ok)
Jul 24 06:52:08 vm1 /usr/sbin/cron[883]: (CRON) INFO (Running @reboot jobs)
Jul 24 06:52:16 vm1 kernel: [ 17.472049] tap0: no IPv6 routers present
Jul 24 06:52:16 vm1 kernel: [ 17.592044] eth0: no IPv6 routers present
Jul 24 06:55:36 vm1 kernel: [ 217.497357] kjournald starting. Commit interval 5 seconds
Jul 24 06:55:36 vm1 kernel: [ 217.497561] EXT3 FS on sda8, internal journal
Jul 24 06:55:36 vm1 kernel: [ 217.497564] EXT3-fs: mounted filesystem with ordered data mode.
==> /var/log/kern.log <==
Jul 24 06:52:07 vm1 kernel: [
Jul 24 06:52:07 vm1 kernel: [
<maxk@qualcomm.com>
Jul 24 06:52:07 vm1 kernel: [
Jul 24 06:52:07 vm1 kernel: [
Control: RX
Jul 24 06:52:07 vm1 kernel: [
ready
Jul 24 06:52:16 vm1 kernel: [
Jul 24 06:52:16 vm1 kernel: [
Jul 24 06:55:36 vm1 kernel: [
Jul 24 06:55:36 vm1 kernel: [
Jul 24 06:55:36 vm1 kernel: [
7.166240] tun: Universal TUN/TAP device driver, 1.6
7.166242] tun: (C) 1999-2004 Max Krasnyansky
7.432019] ADDRCONF(NETDEV_UP): eth0: link is not ready
7.435270] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
7.435927] ADDRCONF(NETDEV_CHANGE): eth0: link becomes
17.472049] tap0: no IPv6 routers present
17.592044] eth0: no IPv6 routers present
217.497357] kjournald starting. Commit interval 5 seconds
217.497561] EXT3 FS on sda8, internal journal
217.497564] EXT3-fs: mounted filesystem with ordered data mode.
Explanation
Prints out current date and time.
Prints last 10 messages from kernel messages buffer.
Prints last 10 messages from /var/log/dmesg, /var/log/messages, /var/log/syslog
and /var/log/kern.log.
132
Exercise 30. Lather, Rinse, Repeat: The
Grand Rote Learning
Amount of information in this guide is pretty big. You cannot retain it without long enough
practice and some hardcore grinding. So the only task left for you will be filling in this table,
and imprinting this table in your memory every day until you know it.
You may want to ask why do you need to memorize all this stuff if you can always look it up.
Well, the short answer will be because you can not. What this means is that to look up things
efficiently you already need what to look for, and to know what to look for, you need a solid
foundation. Once you have this foundation, once you understand what is important and what
is not, and how the system is organized, you will be able to efficiently look things up.
You may wonder why in my guide there are many detailed tables containing lists of many
fields which contain information which you will almost never need. What you must
understand, though, is that you should train yourself to look any console program this way.
You should approach to this information not like to a fiction book, where you may not pay
attention do details and still understand it pretty well. You should approach all those data as
mathematical formulas, where every single symbol has its meaning, and even more so, if
you do not understand what particular symbol means, you will not be able to go further.
Sometimes it is perfectly okay to left something unexplained, but make yourself a present of
going deeper, even if it will be every so often. Give yourself a present by making research
about this particular utility, understand what it tells you and why. If you will do this, if you will
look deep inside, your understanding of an Operating System (Linux in our case) will
increase dramatically.
Documentation
man, info
Command or concept
Meaning
________________________________
______________________________________
man
info
man 1
man 2
man 3
man 4
man 5
man 6
man 7
man 8
man 9
man -k
man -wK
bold text</code>
italic text</code>
[]
-a | -b
argument ...
[expression] ...
Google and useful resources
133
Search
Meaning
term/resource__________________________ __________________________________
_____
____
(a|b) c
site:foo.bar
"a long query"
http://en.wikipedia.org
http://stackexchange.com/
http://www.cyberciti.biz/
http://tldp.org/
programname.site
Package management: Debian package management utility aptitude
Command or concept
Meaning
________________________________
______________________________________
aptitude
aptitude search
aptitude install
dpkg -l
dpkg -L
Desired action
Package status
http://www.debian.org/distrib/packages
System boot: runlevels, /etc/init.d, rcconf, update-rc.d
Command or concept
Meaning
________________________________
______________________________________
rcconf
update-rc.d
sysv-rc-conf
runlevel
runlevel 1
runlevel 2
runlevel 6
Processes: Working with proccesses, ps, kill
Command or concept
Meaning
________________________________
______________________________________
ps
kill
ps ax
ps axue
ps axue --forest
signal
HUP
TERM
KILL
Why KILL -9 is bad?
134
Job chedulers: cron, at
Command or concept
________________________________
crontab -l
crontab -e
crontab -r
crontab /foo
crontab > foo
*****
at
atq
atq
atrm
batch
Logging, /var/log, rsyslog, logger
Command or concept
________________________________
logger
grep -irl
find . -mmin -5
tail -f
logrotate
logging daemon
log level daemon
log rotation
Filesystems
Command or concept
________________________________
Filesystem
File
Directory
Inode
Block
Mounting
UUID
Journal
MBR
Partition
Partition table
Mounting, mount, /etc/fstab
Command or concept
________________________________
parted
cfdisk
fdisk
Meaning
______________________________________
Meaning
______________________________________
Meaning
______________________________________
Meaning
______________________________________
135
mount
umount
mount -a
/etc/fstab
fsck
blkid
logging daemon
log rotation
Creating and modifying filesystems, mkfs, tune2fs
Command or concept
Meaning
________________________________
______________________________________
tune2fs
mkfs
Block size
Reserverd block count
Maximum mount count
Check interval
Changing root directory, chroot
Command or concept
Meaning
________________________________
______________________________________
chroot
ldd
Root directory
Chroot
Dynamic Dependency
Moving data arount: tar, dd
Command or concept
Meaning
________________________________
______________________________________
tar
dd
losetup
Security permissions, chown, chmod
Command or concept
Meaning
________________________________
______________________________________
chmod
chown
umask
Permission
Permission mode
Permission class
Umask mechanism
Networking
Network conecept
Meaning
__________________________________ ______________________________________
OSI model
DOD model
136
Communications protocol
Ethernet
MAC address
Ethernet boradcast address
TCP/IP
IP
IP packet
IP address
IP subnet
Port
Network socket
Local socket address
Remote socket address
Socket pair
Routing
Default gateway
IP boadcast address
ICMP
TCP
TCP packet
UDP
UDP packet
Hostname
Interface configuration, ifconfig, netstat, iproute2, ss
Command or concept
Meaning
________________________________
______________________________________
/etc/network/interfaces
auto
allow-hotplug
/etc/hosts
/etc/hostname
localhost
loopback interface
Pseudo-interface
Configuration files, /etc/network/interfaces
Command or concept
Meaning
________________________________
______________________________________
/etc/network/interfaces
auto
allow-hotplug
/etc/hosts
/etc/hostname
localhost
loopback interface
Pseudo-interface
137
Packet filter configuration, iptables
Command or concept
Meaning
________________________________
______________________________________
iptables-save
iptables
modprobe
nc
tcpdump
LINKTYPE_LINUX_SLL
Ethernet frame header
IPv4 header
TCP segment
netfilter
iptables
iptables tables
iptables chains
iptables target
Secure shell, ssh, sshd, scp
Command or concept
Meaning
________________________________
______________________________________
ssh
sshd
scp
ssh-keygen
Host keys
Auth keys
Data encryption ciphers
Data integrity algorithms
SSH session key
Performance: getting performance stats, uptime, free, top
Command or concept
Meaning
________________________________
______________________________________
uptime
free
vmstat
top
CPU usage (us,sy,id,wa)
Memory (swpd, free, buff, cache,
inact, active)
Slab allocation
Disk (IOPS, read, write)
Process (PR, NI, VIRT,
RES, SHR, Status)
Kernel: kernel messages, dmesg
Command or concept
Meaning
________________________________
______________________________________
138
dmseg
/var/log/dmesg
/var/log/messages
/var/log/syslog
/var/log/kern.log
Kernel message buffer
139