Ted Deacon
Linux Worksheet 4
Tools and Scripts
Learning about scripting will save you time and effort. In the past people needed to interrogate files to automate pulling bits
of information out the files. We are using bash scripts to do this having a working knowledge of this will help you save having
to do things the long way. Work through these examples and note what the outcome is this will help you when you are
required to something similar.
Redirecting output
Hopefully there's no news in the idea that you can redirect the output of a command into a file, this trivial example sends
the output of the command to list the contents of the /etc/ directory into a file called etc.list
ls /etc/ > etc.list
If the file etc.list does not already exist it is created, if however it does exist, the file is overwritten.
If we use a double angle bracket we can append to the output to a file.
date > etc.list
ls /etc/ >> etc.list
The first line creates, or overwrites the existing, etc.list file so that it contains on the current date.
The second line appends the result of the command to list /etc to the file.
We can also use the angle-brackets syntax to input data into a command from a file,
mail -s “Linux Rocks” w.gates@microsoft.com < Linux_Rocks.txt
Here we send an email to Mr Gates, the body of the message will be the contents of the file called Linux_Rocks.txt
Pipes
It should also come as no surprise that we can pipe the output of one command into another,
ls -l /usr/bin | more
The output from listing the /usr/bin directory is piped into the command more, which formats the output so that it displays
one screenful at a time.
Although this might be considered a trivial example, there are some 1870 files 1 in /usr/bin – check it out with ls /usr/bin |
wc -w which will give you a word count of the number of files.
grep
We met grep during several of the previous sessions, you will doubtless recall we used it to check the pids of various
processes,
ps -aux | grep inetd
Ted Deacon
Linux Worksheet 4
The command works by running ps aux -which returns a list of all the processes running on the system and piping the output
through grep which returns only those lines containing the expression inetd.
This combination of commands often produces two lines of output,
root@riktor:~$ ps -aux | grep inetd
80 ?
S
0:00 /usr/sbin/inetd
193 pts/0
S
0:00 grep inetd
To remove the second line of output we could pipe the output through grep again, but this time with the -v switch which
returns lines not matching a given expression,
root@riktor:~$ ps -aux | grep inetd | grep -v grep
80 ?
S
0:00 /usr/sbin/inetd
This may seem like a lot of bother to go to just to exclude one line of a two-line output, but it comes into its own inside a
script where the system would lack the discretion to exclude the irrelevant line of output.
We can also use grep on a file,
root@riktor:/tmp# grep student /etc/passwd
student:x:1000:100:,,,:/home/student:/bin/bash
dopey::0:0:students dodgy root shell:/root:/bin/bash
horrid::0:0:another dodgy root shell by student:/root:/bin/bash
which returns any line that contains “student”.
The -w switch for grep will return all lines that match the exact word specified,
root@riktor:/tmp# grep -w student /etc/passwd
student:x:1000:100:,,,:/home/student:/bin/bash
horrid::0:0:another dodgy root shell by student:/root:/bin/bash
But, we still get a line returned with “student” in the user details section, we can use a caret ( ^ ) to make sure we only get
the lines with an expression at the beginning of the line.
root@riktor:/tmp# grep ^student /etc/passwd
student:x:1000:100:,,,:/home/student:/bin/bash
To match a pattern at the end of a line use the syntax <pattern>$ with grep
grep bash$ /etc/passwd
which will return all the users with a shell of /bin/bash from the password file.
We can also use grep on a whole directory,
root@riktor:~$ grep -r sendmail /etc/rc.d/*
/etc/rc.d/rc.M:# Start the sendmail daemon:
/etc/rc.d/rc.M:if [ -x /etc/rc.d/rc.sendmail ]; then
/etc/rc.d/rc.M: . /etc/rc.d/rc.sendmail start
Which returns all the lines from any of the files in /etc/rc.d/ that contain the expression “sendmail.” Note the “-r” switch
Ted Deacon
Linux Worksheet 4
which tells grep to recurse through any subdirectories, it is, of course, redundant in this example as there no subdirectories
beneath rc.d/ in Slackware
Note also that the name of the file where the results were found is also shown in the output.
sed
We are all familiar with the idea of editing a file in a text editor, but how about editing it on the fly as a stream? Sed is the
stream editor, it usually takes lines of text from standard input and passes them to standard output but performing editing
actions on the way. But it can also take input from a file of from the output of another command,
sed 's/^telnet/#telnet/' /etc/inetd.conf
in this example sed is instructed to substitute any occurrence of telnet at the beginning of a line with #telnet thus
commenting out the telnet line in inetd.conf.
This example would not disable telnet by itself but it could be used as part of a script to perform that task.
As another example, perhaps I just wrote a PHP script and managed to declare my variable “counter” (spelled as it is there
with a lowercase 'c' , but refer to it throughout the remainder of the script with an uppercase “C” as in Counter.
sed 's/Counter/counter/g' counter.php > counter.new
the switch 'g' makes sure that sed replaces all occurrences of Counter, even if there is more than one on a line. The output is
directed into the file counter.new
I would then use cp counter.new counter.php to replace my file with the corrected version.
cut
Cut is used to extract columns or fields from a file.
The fields in the passwd file are delimited by a colon “:” for example,
student:x:1000:100:,,,:/home/student:/bin/bash
the fields are as follows,
1.
2.
3.
4.
5.
6.
7.
user name
Password required
user id (uid)
group id (gid)
“personal” details – full name, room number, telephone extension
path to user's home directory
path to user's shell
To retrieve a list of just the user names from /etc/passwd we could use the following cut command,
root@riktor:~# cut -f1 -d: /etc/passwd
root
.
.
.
nobody
student
relicnet
student
Ted Deacon
Linux Worksheet 4
The arguments to cut are,
-f1
- the first field
-d:
- the delimiter – in this case the colon.
tr
This is the command to translate or delete particular characters from standard input, the results are written to standard
output.
root@riktor:~# echo "this that and the other" | tr " " _
this_that_and_the_other
echo is a command that simply repeats whatever follows, in this example it is piped into tr which translates all instances of a
space ( “ “ ) into an underscore “_” . (Useful when converting Windows style file names with spaces into something more
sensible ;-)
sort
Sort does exactly what it says on the tin, it sorts the lines of text in a file into order. sort -r will sort the lines into reverse
order. Try it out on something simple like the .rhosts file.
If you need to sort the contents of a file and direct it back into the same file use the -o switch,
sort .rhosts -o .rhosts
do not just do this, sort .rhosts > .rhosts - you will lose the entire contents of the file.
uniq
uniq will remove duplicate entries from a sorted file. Consider my clumsily constructed .rhosts file,
root@riktor:~# cat .rhosts
weatherwax
riktor
magrat
petulia
weatherwax
petulia
If we sort the file and pipe the output through uniq the result is a sorted .rhosts with no duplicate entries,
root@riktor:~# sort -r .rhosts | uniq
weatherwax
riktor
petulia
magrat
(If we just run uniq on the file before it is sorted, it will fail to detect and remove the duplicate entries.)
Compressed Archives
Where Windows users have WinZip (and more recently Windows' own built in compression application,) Linux users have
gzip to compress files and archives and gunzip to decompress them.
However it is usual for the files also to be “tarred.” Tar is the GNU version of the tape archiving facility. According to the man
Ted Deacon
Linux Worksheet 4
page for tar it is,
“an archiving program designed to store and extract files from an archive file known as a tarfile. A tarfile may
be made on a tape drive, however, it is also common to write a tarfile to a normal file.”
Tarred and gzipped files are often referred to as a tarball. It is common, for a whole directory, (including any sub-directories,)
to be tarred and gzipped into one archive, - when it is decompressed the original directory structure is recreated.
A tarball will usually have the file extension .tar.gz or .tgz 2
Creating a tarball
To create a tarball of the all the files in your current directory in the file 2013-11-22-myfiles.tgz we would use,
tar -zcpf
2013-11-22-myfiles.tgz *
The options passed to tar are as follows,




z – pass the archive through gzip to compress it
c – create a new archive
p – preserve file permissions and ownership
f – specifies that the next argument is the file name to use for the new file

the final argument is the directory from which to create the archive.
Unpacking a tarball
To unpack the tarball you created,
tar -zxf
2013-11-22-myfiles.tgz
The options passed to tar are as follows,



z – pass the archive through gzip to compress it
x – extract files from the archive
f – specifies that the next argument is the file to be unpacked
Viewing the list of files in an archive
This will simply list the paths/filenames of the files in the archive.
tar -ztf
2013-11-22-myfiles.tgz
cron jobs
crond is the Unix clock daemon that executes commands at specified dates and times according to instructions in a
"crontab" file.
The entries in a crontab file follow this pattern,
2
min
hour
day
10
8
*
month
*
dayofweek
*
command
myscript
You may also come across .zip and .bzip as file extensions, there are usually zip / unzip and bzip / bunzip applications on the Linux system
to handle them.
Ted Deacon
Linux Worksheet 4
which runs myscript at 08:10 every day.
10 8 * * mon myscript
this would run myscript every Monday at 08:10
*/20 * * * * my_other_script
runs my_other_script every twenty minutes, always
date
The Unix / Linux date command sets or prints the date and time from the system clock,
student@weatherwax:~$ date
Sun Dec 7 14:29:21 GMT 2013
Date will also output different formats depending on the options passed, such as
%B – full month name
%d – day of month (01..31)
%m – month (01..12)
%Y – year (1970...)
%s – seconds since 00:00:00, Jan 1, 1970 – Unix time
For example,
student@weatherwax:~$ date +%d.%m.%Y
07.12.2013
Variables
In addition to using the built in commands and utilities provided by the shell, shell scripts may also have variables, control
structures, arrays and many of the other features one might expect of a programming language. (But not the complex
programming features such as objects, multi-dimensional arrays, linked lists.... if you need those it's probably easier to use a
different language, like C,C++, Java, PHP, or Perl)
Variables are declared and assigned values with the syntax, variable_name=value
and they are referenced with, $variable_name
#!/bin/sh
#
msg="hello :-)"
echo
echo $msg
Variables may also be the result of the evaluation of an expression,
#!/bin/sh
Ted Deacon
Linux Worksheet 4
#
dt=`date +%d-%m-%Y`
echo
echo $dt
Note the back quotes ` ` around the date expression which cause it to be evaluated in its entirety before its value is assigned
to the dt variable.
Note also that there are no spaces in either of the examples when assigning a value to a variable.
User Input
Variables may also be taken from user input,
#!/bin/sh
#
echo
echo -n “Please enter your name > “
read name
echo
echo “Hello $name pleased to meet you"
echo

echo -n prevents the output from moving to a new line, in this case it leaves the prompt right after the angle
bracket, waiting for the response.

read is the command that reads the input from the user, the script simply waits until the enter key is pressed, if
anything was entered at the prompt it will be assigned to the variable name.

The echo commands with no arguments simply print a blank line, they are used here to separate the script output
from the shell prompts intending to make the output (slightly?) easier to read.
if... then... else...
#!/bin/sh
#
hr=`date +%H`
period="Morning"
if [ $hr -ge 12 ]; then
period="Afternoon"
fi
if test $hr -ge 18 ; then
period="Evening"
fi
echo
echo “Good $period!"
echo


The hr variable is the current hour of the day, evaluated from an option to date as described earlier.
The period variable is initialised as “Morning” and is changed if either, (or indeed both,) of the following tests is
true.

Note the two different possibilities for the syntax of the if statement.
Ted Deacon

Linux Worksheet 4
The if statements are closed with fi
checking user input
#!/bin/sh
#
echo
echo -n “Would you like to see the fortune cookie (y/N) > “
read ans
if [ “$ans” = “y” ] || [ “$ans” == “Y” ]; then
echo
fortune -o
else
echo
echo “ok no fortune cookie"
fi
echo
 The " (y/N) " in the prompt is a convention used to suggest that “No” would be the default answer, ie. if you just hit
enter, and that you must explicitly answer “y” if you want to go ahead.
The script copes with the case of an answer of “Y” by testing for both “y” OR “Y” - note how “$ans” is also
enclosed in quotes in the test.

Contrary to many other languages, the comparison operators “ = “ and “ == ” both seem to work the same way
here.

The option -o for the fortune command prints a potentially offensive fortune cookie –

You have been warned.
loops
for loop
Bash does not provide quite the same syntax to control its for loops as some of the C-like languages, but we can get close
with,
for (( i=0; i<=10; i=$i+1 ))
do
echo $i
done
...or perhaps a little more neatly with, the seq command,
for i in `seq 10`
do
echo $i
done

In this example seq returns a list of numbers between 1 and 10 – the start point of 1 is the default. We could specify
start and end points with, for example, seq 1997 2013.

Backquotes to evaluate the expression as usual.
We can also use a hard-coded list as the control for the loop,
for i in this that and the other
do
echo $i
Ted Deacon
Linux Worksheet 4
done
or perhaps,
for i in this that “and the other”
do
echo $i
done

In the above examples everything after in is taken as an item in the list, the for loop will execute whatever
commands are specified for each item in the list.

In the second example “and the other” is taken as one single item.
We can also generate a list using other command(s),
for i in $(ls)
do
echo $i
done
the above example is the same as,
list=`ls`
for i in $list
do
echo $i
done
The difference between the two previous techniques is minimal at this level, however the second version may be easier to
use when there are more complex commands involved to generate the list.
while loop
With the exception of the slightly unusual bash syntax while loops operate much as you would expect,
declare -i counter=0
while [ $counter -lt 10 ]
do
echo $counter
counter=$counter+1
done


declare -i sets the type for the variable counter to integer
-lt in the test for the while loop means less than. Take care when attempting to use the 'usual' comparison
operators < and > in case bash thinks you are trying to read from a file or direct output into a file.
until loop
until loops work in much the same way as while loops except that they run until the test returns true,
declare -i counter=20
until [ $counter -lt 10 ]
do
echo $counter
#counter=$counter-1
Ted Deacon
Linux Worksheet 4
let counter-=1
done

Note the alternative syntax used to decrement the counter here. (we could equally have used
let counter+=1 in the previous example.)
Command line arguments
A shell script may also accept arguments on the command line,
#!/bin/sh
#
echo
echo “Name of script:
$0“
echo
echo “Hello $1“
echo “you put $# arguments on the command line"
echo
echo “here is a list of them:
$*"
echo
echo “and here they are one by one:"
count=1
for i in $*
do
echo “Argument $count was $i”
count=$[count+1]
done





The arguments are accessed by the variables $0 to $n where $0 is the name of the script itself.
$# returns the number of arguments supplied to the script
$* contains a list of all the arguments.
Note the syntax of the simple for loop that works through the list contained in $*
Note also the use of another alternative syntax to increment the counter, the concise and convenient “counter++”
syntax of C-like languages is not available in bash!
Working example
Finally here's an example of a script that actually does something useful and that I used in anger recently – I received some
ninety-odd picture files to be uploaded to a web site, my client had created them on a windows box and of course had put
spaces in the names. I'm not saying that it was quicker to write this than rename them by hand – but it was most definitely
more fun to do it this way – and it will be quicker next time ! ;-)
#!/bin/sh
#
#mvg.16.09.2013.fix-space-name remove spaces from *@$!^¬* windows file names
#
#translate spaces in filenames into "_" so that $file_list contains
#a space separated list of files
# gives an error if NO files have spaces in the name
file_list=`ls | tr "\ " _`
#
for i in $file_list
do
j=`echo $i | tr _ "\ "`
#get "real" filename (with spaces)
if echo $j | grep " " >/dev/null; then
#if there's a space in the name
Ted Deacon
mv "$j" $i
echo "Renaming $j to $i"
fi
done
Linux Worksheet 4
#rename it with "_" instead of space
#quotes around $j otherwise sh thinks it's a list!
#confirm what's being done
##else don't bother