Introduction to shell scripting
Andreas Buzh Skau
buzh@usit.uio.no
Research Computing Services
Slides: http://folk.uio.no/buzh/bash
RCS Course Week Fall 2015
Outline
◮
What is shell scripting ?
◮
How to run the scripts
◮
Variables
◮
Control flow (tests, if-then-else-fi, loops)
Poll
◮
Who has already
◮
used a shell?
Poll
◮
Who has already
◮
◮
used a shell?
written a shell script?
Poll
◮
Who has already
◮
◮
◮
used a shell?
written a shell script?
written a script (R, MATLAB, . . . )?
Poll
◮
Who has already
◮
◮
◮
◮
used a shell?
written a shell script?
written a script (R, MATLAB, . . . )?
written a program (C/C++, Fortran, Java, . . . )?
What is a shell?
◮
The shell is a computer program that presents you with a
command line interface.
◮
You type in a command or set of commands, hit enter and
the shell does what you ask
◮
A supplement or alternative to a Graphical User Interface
(GUI)
⇒ The shell is made for human interaction with computers
What is a script?
◮
A script is a list of commands that the shell will execute in
order.
◮
It’s kept in a text file, rather than being a single line on the
command prompt
◮
Think of it like the script to a movie or play. You are a
director giving a script to the actor.
◮
The actor reads, interprets and executes the script
according to the director’s instructions.
⇒ The shell does the same with your scripts.
Why shell scripting?
◮
A way to store, edit & execute shell commands
◮
◮
◮
◮
store ⇒ avoid repetitive work
adapt ⇒ easily change commands
logic ⇒ gather information and act accordingly
automate ⇒ computers are here to work for you
⇒ Simplifies the repetition of long & complex command
sequences.
What is shell scripting used for?1
◮
A wrapper for a program
SLURM job scripts are shell scripts
Start a program with the same options over and over, easily
◮ Prepare job input data, run job, check results, copy results
⇒ Save time by automating
◮
◮
◮
To manage research environment
Create data/jobs layout (e.g., for parameter studies)
Perform post-processing of results
◮ Document exactly what you did
⇒ May help in reproducing results if necessary.
◮
◮
1
on Abel
Example script: simple.sh
◮
use any (ASCII) text editor: vim, emacs, nano, gedit, . . .
#!/bin/bash
# An example script for the RCS course week
WHO="All"
if [ $# -gt 0 ]
then
WHO="$1"
fi
DATE=$(date)
echo "Welcome to the tutorial, $WHO"
echo "$DATE"
Simple script: How to run it
> bash simple.sh
Welcome ’All’ to RCS Course Week
Oct 27 12:50:23 CET 2015
> bash simple.sh Abel
Welcome ’Abel’ to RCS Course Week
Oct 27 12:50:23 CET 2015
◮
Run script without bash ...?
> ./simple.sh
-bash: simple.sh: Permission denied
◮
Check
> ls -l simple.sh
-rw-r--r-- 1 buzh users 117 Oct 22 18:04 simple.sh
◮
Fix
> chmod u+x simple.sh
> ls -l simple.sh
-rwxr--r-- 1 buzh users 117 Oct 22 18:04 simple.sh
Invocation – Put it in the PATH
◮
How about saving two more keystrokes?
> simple.sh
-bash: simple.sh: command not found
◮
Check
> which simple.sh
/usr/bin/which: no simple.sh in (/hpc/bin:...
which searches the directories in the environment variable PATH
> PATH=$PATH:$HOME/scripts
> simple.sh
Welcome ’All’ to RCS Course Week
Oct 27 12:50:23 CET 2015
◮
Alternative: use absolute path, i.e., /path/simple.sh
Hashbang 2
#!/bin/bash
◮
1st line in each shell script
◮
Lets the OS know which interpreter to use for a script
◮
Not mandatory, but recommended
◮
Different shells have different syntax (sh, ksh, tcsh, zsh etc)
◮
Also works for some other languages, like Perl, PHP etc
2
# is the hash, and ! is the bang. Remnants from the printing industry
Comments
◮
Anything preceded by a # (hash) is ignored by the shell
◮
We call this a ”comment”
◮
The hashbang is a special comment
◮
If you wish to disable part of your script temporarily, just
comment it out
# echo "Velkommen til ITF kurs-uke!
echo "Welcome to the RCS course week!"
Arguments – Welcome ’everybody’
◮
Passing arguments
> arguments.sh everybody
Welcome ’everybody’ to RCS Course Week
◮
Simplified script
echo "Welcome ’$1’ to RCS Course Week"
◮
$1 is a positional parameter
◮
◮
positional parameters are set when the shell is invoked
set with everybody in the invocation above
Arguments – Welcome everybody
◮
Passing multiple arguments
> argumentsX.sh Alice Bob
Welcome ’Alice’ (1) to DRC Course Week
Welcome ’Bob’ (2) to DRC Course Week
Welcome ’’ (3) to DRC Course Week
◮
Enhanced script
#!/bin/bash
echo "Welcome ’$1’ (1) to RCS Course Week"
echo "Welcome ’$2’ (2) to RCS Course Week"
echo "Welcome ’$3’ (3) to RCS Course Week"
◮
$n is a positional parameter
◮
unused positional parameters are empty
Variables
◮
Assignment
VARIABLENAME="any string you like"
◮
◮
◮
◮
VARIABLENAME is of characters a-z, A-Z, 0-9, _, . . .
No space between VARIABLENAME and = and value
Quoting (" or ’) needed if value contains spaces
Usage
$VARIABLENAME
◮
◮
no need to declare before use (default: empty string)
Before the shell executes a line, it will expand
$VARIABLENAME to the actual value contained within.
◮
Note the difference (with/out $)
◮
Untyped (no difference between numbers, text etc)
Variables – Examples
◮
Script
#!/bin/bash
ABEL=Abel
ABLE="Abel is
echo "Welcome
echo "Welcome
echo "Welcome
◮
able"
’$ABEL’ (ABEL) to DRC Course Week"
’$ABLE’ (ABLE) to DRC Course Week"
’$UNSET’ (UNSET) to DRC Course Week"
Output
Welcome ’Abel’ (ABEL) to DRC Course Week
Welcome ’Abel is able’ (ABLE) to DRC Course Week
Welcome ’’ (UNSET) to DRC Course Week
◮
Experiment with quoting
◮
◮
remove double quotes in assignment of ABLE
remove single and/or double quotes in echo for UNSET
Variables – Examples (2)
◮
Variables
mypath=/tmp/scripting_course
myfilename=magic_script.sh
myfile=$mypath/$myfilename
mytext="Welcome !"
mynumber=99
myarray=(apple banana strawberry)
◮
Use
mkdir -p $mypath
echo "$mytext" > $myfile
echo $mynumber >> $myfile
mv $myfile $myfile.${myarray[2]}
for ((i=0; i<=$mynumber; i++)); do echo $i; done
Status
◮
You should now understand the highlighted parts.
#!/bin/bash
WHO=All
if [ $# -gt 0 ]
then
WHO=$1
fi
DATE=$(date)
echo "Welcome ’$WHO’ to RCS Course Week
echo "$DATE"
Variables – Command substitution
◮
The ”date” commands returns the date. We could
timestamp our output like this:
#!/bin/bash
echo -n "Work start at "; date
sleep 3 # Placeholder for actual work
echo -n "Work finished at "; date
Work start at Thu Apr 16 14:28:33 CEST 2015
Work finished at Thu Apr 16 14:28:36 CEST 2015
◮
But what if we wanted to save the date ouput for later use?
Variables – Command substitution(2)
◮
Syntax
VARIABLENAME=$(command with arguments)
◮
Variable is substituted with the command’s output.
#!/bin/bash
START=$(date)
sleep 3 # Placeholder for actual work
STOP=$(date)
echo "Work started at $START"
echo "Work finished at $STOP"
Work started at Thu Apr 16 14:48:49 CEST 2015
Work ended at Thu Apr 16 14:48:52 CEST 2015
◮
Put any command output into a variable!
Control flow
◮
Automate decision making
◮
Adapt to different conditions
◮
Scripts are more than a list of sequential commands
If-then-else – Syntax
if condition
then
some_commands
else
some_other_commands
fi
◮
else branch is optional
◮
condition can be any command!
A quick note about exit codes
◮
All commands in Linux/unix return an exit code
◮
A command that finishes succesfully returns 0
◮
Any non-zero exit code indicates failure
◮
The exit code of the previous command is stored in the
special variable $?
⇒ ”if” responds to the exit code of the condition
If-then-else – Example
◮
Example
#!/bin/bash
CARROTS=10
HORSES=8
if test $CARROTS -ge $HORSES
then
echo "There are enough carrots"
else
echo "Warning: Not enough carrots!"
SADHORSES=$(expr $HORSES - $CARROTS)
echo $SADHORSES will not get a carrot.
fi
There are enough carrots
test – It’s a command called test
◮
For details see man test
◮
Examples
expression
evaluates to true if
EXP1 -a EXP2 both EXP1/2 are true
EXP1 -o EXP2 either EXP1/2 is true
INT1 -eq INT2 integers are equal
INT1 -ge INT2 INT1 ≥ INT2
-e FILE FILE exists
◮ Square brackets in bash is an alias for ’test’
◮
Can improve readability
if [ $CARROTS -ge $HORSES ]
Subshells
◮
Multiple commands can be grouped together within
parenthesis.
if (command; othercommand && ... )
then
echo "Success!"
fi
◮
Run any number of commands, but make sure you
understand the final exit status
Subshells - Final exit status
if (ls /TMP; ls /tmp)
then
echo "Success!"
fi
◮
The last command returns 0, so it’s a success, despite the
first command failing
if (ls /TMP && ls /tmp)
then
echo "Success!"
fi
◮
This would NOT return ”Success!”
Subshells - Using the output
echo $(date)
DATE=$(date)
echo $DATE
◮
Preceding a (subshell) with a $ will return the output of the
executed command
Loops
◮
Loops can do the same thing over and over
◮
Two main types of loops: ”for” and ”while”
(for) – loop syntax
for VAR in LIST
do
work goes here
done
◮
VAR is set to the first item of the list, then the work is done.
(no $)
◮
LIST can be given like ”apple cherry kiwi” or as output of a
program. E.g. $(cat fruits.txt)
◮
Once the work is done for the first item, VAR is set to the
next item in LIST and so on, until LIST is empty
Loops – Example
for NAME in Alice Bob Cindy Dave Erica Frank
do
echo "Hello $NAME"
done
for NAME in $(cat names.txt)
do
echo "Hello $NAME"
done
(While) – loops
#!/bin/bash
i=1
while (( i <= 10 ))
do
echo "I can count to $i"
i=$(($i+1))
done
◮
While loops start only if the expression is true, but continue
until it becomes false
Become a shell scripting user/expert
◮
Read man pages: man bash, man test
◮
Advanced Bash-Scripting Guide:
http://www.tldp.org/LDP/abs/html/
◮
Learn to use cmdline tools like grep, awk, sed, ...
◮
Study examples
◮
Online search
◮
Most important! Practice, practice, practice.
Happy scripting
◮
Course by Andreas Buzh Skau
◮
buzh@usit.uio.no
◮
Slides: http://folk.uio.no/buzh/bash/
Tools, tips and tricks
◮
Debug output
[buzh@stridselg ˜]$ cat debug.sh
#!/bin/bash
cd /tmp
if [ -f test ]; then
ls -l test
else
echo Nope
fi
[buzh@stridselg ˜]$ bash debug.sh
Nope
[buzh@stridselg ˜]$ bash -x debug.sh
+ cd /tmp
+ ’[’ -f test ’]’
+ echo Nope
Nope
◮
Common pitfall: Invisible characters
[buzh@stridselg test]$ cat invisible.sh
one=1
two=2
e c h o o n e plus two equals $(( $one + $two ))
[buzh@stridselg test]$ cat -v invisible.sh
one=1
two=2
echoM-BM- one plus two equals $(( $one + $two ))
◮
This script would give an error: ”command not found”
◮
The reason is that the character between ”echo” and ”one”
is not a space, even though it looks like it
◮
This particular one comes from accidental Alt-Space
instead of Space
◮
’cat -v’ displays non-printing characters in files
◮
Search text with ”grep”
$ cat debug.sh | grep test
echo test
$ grep test debug.sh
echo test
$ grep -c test debug.sh
1
◮
’grep’ is purpose built to search text. Very powerful!
◮
’man grep’ has all the options, but can be daunting.
◮
Display part of a string with ’cut’
$ grep nobody /etc/passwd
nobody:x:99:99:Nobody:/:/sbin/nologin
nfsnobody:x:65534:65534:Anonymous NFS User:/var/
$ grep nobody /etc/passwd | cut -d: -f5
Nobody
Anonymous NFS User
◮
-d declares the delimiter, -f the field
◮
If you have advanced needs, look at ’awk’ instead
◮
Change text with sed
$ grep nobody /etc/passwd
nobody:x:99:99:Nobody:/:/sbin/nologin
nfsnobody:x:65534:65534:Anonymous NFS User:/var/
$ grep nobody /etc/passwd | sed ’s/x/y/g’
nobody:y:99:99:Nobody:/:/sbin/nologin
nfsnobody:y:65534:65534:Anonymous NFS User:/var/
◮
The command above is ”swap x with y for all lines”
◮
Forward slashes delimit the components of the command
◮
Powerful: Whole books have been written about it
The end!