Korn Shell Script Writing & HPC
Science & Technology Support Group
High Performance Computing
OSC (Ohio Supercomputer Center)
1224 Kinnear Road
Columbus, OH 43212
1
Table of Contents
•
Introduction
–
–
–
–
–
•
What is a shell script?
Script Programming Language
Why use shell scripts?
Korn shell scripts
References
Sampling of Script Features
A simple script
A more versatile Script
Creating Files within a Script
A documented Script
Script Basics
– Creating a shell script
– Executing a shell script
– Echoing script commands
Script Variables
– User-defined Variables
– Storing a Value into a Variable
–
–
–
–
–
–
The Evolution of a Script
–
–
–
–
–
•
•
•
• Assignment Operator =
• Read Command
• Command Substitution
Null Variables
Special Shell Variables
Positional Parameters
Referencing Variables
Parameter Substitution
Exercises
Decision-Making statements
–
–
–
–
–
–
–
–
Flow Control
Exit Status
If-Then-Else command
Conditional Tests
Logical Operators
Illustrative Examples
Case statement
Exercises
2
Table of Contents
•
Looping Statements
–
–
–
–
–
•
Arithmetic Operations
Operator Precedence
Numerical Bases
String vs Integer Variables
Exercises
Script I/O
–
–
–
–
–
–
Shell Redirection Operators
"Here-is" Files
print
read
Quoting
Exercises
Communicating with Running
Scripts
– Sending Signals
– How Scripts Receive Signals
•
Working with Integers
–
–
–
–
–
•
for Statement
select Statement
while Statement
Until Statement
Exercises
•
•
Script Functions
–
–
–
–
–
Why Use Functions?
Syntax & Use
Auto-Loading
Function Definition Location
Command Precedence
HPC Designed Scripts
–
–
–
–
Timing Loops
Debugging Options
Batch Jobs
Performance Tool
Preprocessing
– Multiple Tools runs
– Multiple Tool Reports
– File Renaming
3
Introduction
•
•
•
•
•
What is a shell script?
Script Programming Language
Why use shell scripts?
Korn Shell scripts
References
4
What is a shell script?
•
•
•
•
A shell script is a collection of Unix commands entered into
a file. When the file (script) is executed, the commands are
executed in order.
Running a shell script is EXACTLY equivalent to
interactively entering each command
For example, say a user always does the same four steps
before printing out a file: making the file owner- writable,
using vi to check/change the file, printing it out, and then
making the file unwritable (for safety reasons)
An interactive session of a user following this procedure
for the file comp.txt is shown on the next page
5
Interactive editing and printing
$ chmod u+w comp.txt
$ vi comp.txt
Here is a table summarizing OSC computers as of today.
Supercomputer
# Processors
Proc Speed
-----------------------------Itanium Cluster 124
900 MHz
Cray SV-1
16
300 MHz
Origin 2000
64
300 MHz
IA-32 Intel
128
550 MHz
Sun Fire 6800
24
900 MHz
"comp.txt" 11 lines, 535 characters
$ lpr -Pps1 comp.txt
Ideal Calc.
------------223.2 GFLOPS
19.2 GFLOPS
38.4 GFLOPS
70.4 GFLOPS
43.2 GLOPS
$ chmod u-w comp.txt
6
Making the viprint script
•
•
•
To make the previous editing and printing procedure into a
script, all the user has to do is type the same commands
they use interactively into a file
Say the file is called viprint, it looks like this:
$ cat viprint
chmod u+w comp.txt
vi comp.txt
lpr -Pps1 comp.txt
chmod u-w comp.txt
After viprint is made executable, whenever the user wants
to execute these four commands they simply type viprint as
if it was a “normal” Unix command
$ viprint
7
Script Programming Language
•
•
The real power is writing shell script comes from using
special shell commands and variables that essentially make
your script act like a program
In this course, you will learn about the programming
statements which can be used in scripts, including:
– Initializing and using your own script variables. Both single valued
and arrays
– Performing operations with variables
– Decision-making statements (if-then-else, etc)
– Looping statements (for,etc)
– Defining and using functions in scripts
– Script I/O commands and procedures
•
•
Above bullet reads like a “table of contents” for a text on a
“traditional” programming languages such as Fortran or C,
although …
Script programming has less capabilities and commands
8
Why use shell scripts?
•
•
•
•
•
•
•
•
•
Convenience: Run a number of Unix commands by typing a single
script name
Command log: Script can permanently contain a detailed set of
commands with numerous options. (If interactive, complex command
is gone)
Usefulness: Write a script with programming commands to give the
script a number of capabilities/options
Take “full” use of what the shell has built-in to offer you
Considering a script as a program, no compiler, loader, or libraries
needed to make it executable
Portability: If a script works in the shell on your machine it will work
on the same shell on a different machine
Easy to debug: Small number of programming statements which are
relatively simple
Easy to modify, expand, and customize: Just type in the new
commands
Run-type interaction: Can send signals to running scripts
9
Korn shell scripts
•
Several types of Unix shells exist, each with their own script
programming languages:
– Bourne (sh), Born Again shell (bash), Korn shell
– C shell (csh), TENEX C shell (tcsh)
•
Features you would expect from a well-written shell:
– Convenient interactive features: command history, command-line editing,
and filename completion
– Powerful scripting capabilities including reliability and efficiency
•
This course will discuss Korn shell scripting
– It has all the convenient interactive features
– It is backward compatible with the Bourne shell (the first shell)
• Most scripts are sh
• Start-up sequence for ksh same as sh only enhanced
• Unix utilities and administration commands depend on sh
– It does not have the scripting bugs that the C shell has
10
Korn shell scripts
•
What if I already use a different shell on my machine?
– Could switch to Korn shell (use the chsh command if you can)
– Keep your existing shell as your interactive shell and write your scripts as
Korn shell scripts (Easily done: one line needed in the script)
11
References
•
Internet URLs
– nacphy.physics.orst.edu/rubin/melanie/node144.html (Tutorial)
– gonzo.tamu.edu/csh.whynot.html (Bugs in C shell scripting)
•
Usenet Newsgroups
– comp.unix.shell
– comp.unix.questions
•
Anonymous ftp
– athos.rutgers.edu/~patra/vi.html (shell-101.BetaA.Z)
– rtfm.mit.edu:/pub/usenet/news.answers/unix-faq/faq (Unix FAQ)
•
OSC Unix-Series workshops (oscinfo.osc.edu)
– Basic Unix
– Intermediate Unix
– Introduction to Perl
12
The Evolution of a Script
•
•
•
•
•
Sampling of Scripting Features
A Simple Script
A More Versatile Script
Creating Files within a Script
A Commented Script
13
Sampling of Scripting Features
•
Next few slides will act as a course overview as we take a simple
script and keep extending its capabilities
•
The more advanced scripts will demonstrate some of the
capabilities of Korn Shell scripting
•
Don’t worry about understanding all the details at this point. A
complete explanation of the scripting commands used is given in
later chapters
14
A Simple Script
•
•
In the script called triangle, a C program is compiled and then
executed. The program calculates the area of the triangle and
reads in the lengths of the sides from a file and outputs the result
to another file.
Here is an example of a user running the script:
$ cat triangle
cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o area area.c
area < sides.in > area.dat
cat area.dat
rm area
$ ls -l triangle
-rwxr--r-1 dje
appl
108 May 3 12:09
triangle
$ cat sides.in
8 3 7
$ triangle
Area of Triangle is 10.392305
15
A Simple Script
•
Limitation of the simple triangle script
–
Script only works for the one area program. What if you want to run
other triangle-related programs with this same script ?
16
A More Versatile Script
•
•
Scripts can take arguments on the command line when they are
run. The first argument is passed to a variable with a special
name: $1. We can use this feature to make the triangle script work
for any program.
Here is the resulting new script in action:
$ cat triangle
cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c
$1 < sides.in > $1.dat
cat $1.dat
rm $1
$ triangle area
Area of Triangle is 10.392305
$ triangle perim
Perimeter of Triangle is 18.000000
17
A More Versatile Script
•
Limitation of versatile script:
– Scripts should be as independent of possible of “outside” existing files (for
portability and ease of use). Since the sides.in file is so short, we should
create the file within the script
18
Creating Files Within a Script
•
The easiest method for creating a small “internal” file is to use
“Here-is” Unix redirection. After it is used, this temporary file is
removed:
$ cat triangle
if [[ $2 = debug ]]; then
cc -g -o $1 $1.c
print "$1 meant to be debugged"
else
cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c
cat << EOF > sides.in
8 3 7
EOF
$1 < sides.in > $1.dat
cat $1.dat
rm $1 sides.in
fi
$ triangle perim
Perimeter of Triangle is 18.000000
19
A Commented Script
•
•
•
A bad script writing style has been shown in all
incarnations of the triangle script: the lack of comments.
Comments are critical to another user (or you) in
understanding what the script does.
The special “comment” on the first line of script actually
specifies the shell to be used for the script commands. This
is required for users whose interactive and script shells
differ.
On the next page is the timing script just discussed but
now with proper commenting
20
A Commented Script
$ cat triangle
#!/bin/ksh
# Compiling C code found in first parameter
cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c
# Using redirection to make input file
cat << EOF > sides.in
8 3 7
EOF
# Using while loop to run the executable 17 times
let i=0
while (( i < 17 )); do
$1 < sides.in >> $1.dat
let i=i+1
done
# Checking output
cat $1.dat
# Clean up files no longer needed
rm $1 sides.in
21
Script Basics
•
•
•
Creating a Shell Script
Executing a Shell Script
Echoing Script Commands
22
Creating a Shell Script
•
A shell script is an executable file which is executed by the
shell line-by-line. It can contain the following:
– UNIX commands
– shell programming statements
– comments
•
•
Create using editor of choice
Can include a #! construct in first line of script to override
login shell
– #!/bin/ksh uses Korn shell to execute script
– #!/bin/csh uses C shell to execute script
– Recommend always put this in every Korn shell script. Makes it
clear what the file is. "file" Unix command will even identify it as a
Korn Shell script
•
Name should be short and descriptive
23
Executing a Shell Script
There are 3 ways to execute a shell script:
1."dot" method
$ . scriptname
2."just the name" method
$ scriptname
3.in the background
$ scriptname &
24
Executing a Shell Script
•
Method 1 runs the command as if you typed them in on the
command line
– No ksh subprocess created
•
Note that methods 2 and 3 require:
– execute permission for scriptname
chmod +x scriptname
– current directory (.) must be in PATH or else must use
./scriptname
25
Executing a Shell Script
•
Demonstration of execution methods:
$ ps
PID TTY
31612 ttyq5
31631 ttyq5
$ ls -l showsh
-rw-r--r-1 dje
$ cat showsh
print Executing the
ps
$ . showsh
Executing the shell
PID TTY
31612 ttyq5
31628 ttyq5
TIME CMD
0:00 sh
0:00 ps
appl
43 Nov 25
1997 showsh
shell script showsh
script showsh
TIME CMD
0:00 sh
0:00 ps
26
Executing a Shell Script
$ chmod u+x showsh
$ showsh
Executing the shell
PID TTY
31578 ttyq5
31612 ttyq5
31635 ttyq5
script showsh
TIME CMD
0:00 ps
0:00 sh
0:00 sh
27
Echoing Script commands
•
•
•
•
•
For understanding or debugging scripts it is often
convenient to see the script commands shown on the
monitor along with the output they produce.
Accomplished using the set -x command at the beginning
of the file
To turn off the echoing, make the command set +x (counterintuitive?)
Also useful in batch job files and - on occasion interactively
On the next page a demonstration of the power of set -x is
shown
28
Echoing Script commands
$ see
/homea/dje/ksh_script
boba.osc.edu:0.0
a_opt.txt
cba_opt.txt
b_opt.txt
fant4.txt
c_opt.txt
item.intmod.txt
$ cat see
#!/bin/ksh
pwd
echo $DISPLAY
ls *.txt
$ cat see2
#!/bin/ksh
set -x
pwd
echo $DISPLAY
ls *.txt
item.set.3.txt
item.set.4.txt
no_opt.txt
pid.txt
showsh.txt
trap.txt
29
Echoing Script commands
$ see2
+ pwd
/homea/dje/ksh_script
+ echo boba.osc.edu:0.0
boba.osc.edu:0.0
+ ls a_opt.txt b_opt.txt c_opt.txt cba_opt.txt
fant4.txt item.intmod.txt item.set.3.txt item.set.4.txt
no_opt.txt pid.txt showsh.txt trap.txt
a_opt.txt
cba_opt.txt
item.set.3.txt
pid.txt
b_opt.txt
fant4.txt
item.set.4.txt
showsh.txt
c_opt.txt
item.intmod.txt no_opt.txt
trap.txt
30
Script Variables
•
•
User-defined Variables
Storing a Value into a Variable
– The assignment operator =
– The read command
– Command Substitution
•
•
•
•
•
•
Null Variables
Special Shell Variables
Positional Parameters
Referencing Variables
Parameter Substitution
Exercises
31
User-defined Variables
•
The Korn shell includes the following types of variables:
– User-defined variables
– Special shell variables
•
User-defined variables can be initialized, used and changed
from the command line or from within a shell script.
– Not "declared" like in a programming language. Just type the name
of the variable
– Korn shell scripts use string and integer variables
•
A variable name can consist of the following:
– letters, digits, and the underscore character
– first character of a variable name must be a letter or an underscore
character
•
A variable can be made read-only: once it is assigned a
value, the value cannot be changed
– "Archaic" method: readonly variable_name
– Alternate method: typeset -r variable_name
32
Storing a Value into a Variable
• There are several methods for variables to obtain values.
 The assignment operator =
• Enter the name that you have chosen for the variable
followed by an equal sign and then the value that you want
to store in the variable.
• To use the a variable's value, put the $ operator in front of
it's name. (Read the $ as "contents of")
• The typeset command can also be used for variable
assignment. More up-to-date procedure, typeset has many
options.
• To assign a value to an integer variable, the assignment
command must be proceeded by the word let
• On the next page, a script is shown that demonstrates all
these approaches
33
Storing a Value into a Variable
$ cat assign
street=Elm
print street
<-- Most common error when using variables
print $street
my_name="John Smith"
echo $my_name
neph=Chris Sellgren
print $neph
typeset card=ace; print $card
let num=57; echo $num
let num=num+3; print $num
$ assign
street
Elm
John Smith
assign2.ksh[9]: Sellgren: not found
ace
57
60
34
Storing a Value into a Variable
 Second method: read in a value for a variable
•
So far have used print and echo (archaic) commands for
printing out all the strings following them and then a
carriage return
•
There is a complimentary command read which input a
value into variables. The user type in as many strings as
there are variables on the read line, and then hits carriage
return
•
On the next page is a demo script indicating how the read
command works.
35
Storing a Value into a Variable
$ cat input
print Please enter a ship name
read ship
print Please enter your full name
read fname lname
print Captain $lname, welcome aboard the $ship
echo May I call you $fname?
print Enter an integer; read num
let num=2*num
echo Twice your number is $num
$ input
Please enter a ship name
Redoubt
Please enter your full name
Horatio Hornblower
Captain Hornblower, welcome aboard the Redoubt
May I call you Horatio?
Enter an integer
27
Twice your number is 54
36
Storing a Value into a Variable
 Use command substitution. The string(s) that a Unix
command returns may be put into a script variable
•
The following syntax is as follows:
var_name=$(UNIX command)
•
As with all the script commands, command substitution
can be done interactively as well
•
On the next page is a demo script illustrating this technique
37
Storing a Value into a Variable
136:oscb$ cat command
dir=$(pwd)
print I am in the directory $dir
weekday=$(date +%A)
print It is $weekday
files=$(ls -lt | wc -l)
print There a $files files in this directory
me=$(whoami)
print I am $me
computer=$(hostname)
print My computer is called $computer
logged_on=$(who | cut -f1 -d' ')
print These people are also on $computer: $logged_on
137:oscb$ command
I am in the directory /home/dje
It is Tuesday
There a 277 files in this directory
I am dje
My computer is called oscb
These people are also on oscb: osu2796 opr006 paul ysu039 osu2796 jimg
mmittal jimg mmittal osu2224 osu2224 srb cls038 osu1938 jeff dje
38
Null Variables
A variable can be set to a null value, even if previously
assigned, using any of the following methods. There
should be no spaces preceding or following the equal sign.
If the contents of null character variables are printed out, a
blank appears. If the contents of a “nullled” integer variable
is outputted, a 0 is displayed.
$ name=
$ name=''
$ name="”
$ let num=
$ unset varname
39
Null Variables
All variables that don't exist are assumed null unless set -o
nounset is used. Then the shell will indicate an error when
an undefined variable is encountered
$ unset name
$ set -o nounset
$ print $name
ksh: name: parameter not set
$ set +o nounset
40
Special Shell Variables
•
•
•
In addition to your own user-defined variables, other
variables defined by the shell for the user are called special
shell variables and are most useful
These variables are set and updated automatically by the
shell. Their values cannot be changed, but they may be
referenced.
The special shell variables of most use are identified by the
following symbols:
$#
$$?
$$
$!
$0
$*
41
Special Shell Variables
•
The variable $# contains the number of arguments typed on
the command line.
$ cat numargs
print The number of arguments is $#
$ numargs
The number of arguments is 0
$ numargs 1 2 3 4 5
The number of arguments is 5
$ numargs "Hello World"
The number of arguments is 1
42
Special Shell Variables
•
The variable $- contains the shell flags (options) of the
current shell.
$ print $isum
$ set +u
$ echo $ism
43
Special Shell Variables
•
The variable $? contains the exit status of the last
command. If the exit status is 0, the command ran
successfully. If it is non-zero, there was a problem
$ ls
file1
data
account.txt
$ rm file1
$ print $?
0
$ rm dataa
dataa: No such file or directory
$ print $?
2
•
We will see exit status again. It will be used to define true and
false. Also, the user can set the exit status in their script by
using the exit command
44
Special Shell Variables
•
The variable $$ contains the Process ID (PID) of the
current shell process.
$ cat pid
#!/bin/ksh
ps
print $$
$ pid
PID
34738
34744
34765
34765
$ . pid
PID
34743
34744
34744
TTY
ttyq5
ttyq5
ttyq5
TIME
0:00
0:00
0:00
CMD
ps
sh
pid
TTY
ttyq5
ttyq5
TIME CMD
0:00 ps
0:00 sh
45
Special Shell Variables
•
The variable $! contains the process ID number of the last
command sent to the background.
$ sleeper &
[1]
29144
$ ps
PID TTY
TIME
29144 ttyq5
0:00
29150 ttyq5
0:00
29160 ttyq5
0:00
29167 ttyq5
0:00
$ compress moby_dick &
[2]
29136
$ print $!
29136
CMD
sleeper
sh
ps
sleep
46
Special Shell Variables
•
The variable $0 contains the name of the script currently
being execute
$cat old_script
print The name of this script is $0
$old_script
The name of this script is old_script
$mv old_script new_script
$new_script
The name of this script is new_script
•
The variable $* contains all the arguments on the command
line.
$ cat args
print The arguments are: $*
$ args bob dave
The arguments are: bob dave
47
Positional Parameters
•
•
For a script, the command line arguments are also called
positional parameters. Each positional parameters refers to
one individual argument on the command line. The ten
positional parameters available to the script writer are
referenced as follows:
$1 $2 $3 $4 $5 $6 $7 $8 $9
The parameter $1 contains the first argument, $2 the
second argument, and so on. Consider the following script:
$ cat
print
print
print
print
print
parms
Arg 1
Arg 2
Arg 3
Arg 4
Arg 5
is:
is:
is:
is:
is:
$1
$2
$3
$4
$5
48
Positional Parameters
$ parms 2001 A Space Odyssey
Arg 1 is: 2001
Arg 2 is: A
Arg 3 is: Space
Arg 4 is: Odyssey
Arg 5 is:
$ parms "Space, the final frontier"
Arg 1 is: Space, the final frontier
Arg 2 is:
Arg 3 is:
Arg 4 is:
Arg 5 is:
49
Positional Parameters
Use the set command to change positional parameters. It
replaces existing positional parameters with new values
$ cat newpos
print starting args are $*
print number of args is $#
print arg 1 is $1
print arg 2 is $2
set NCC 1701 Enterprise
print new args are $*
print number of args is $#
print arg 1 is $1
print arg 2 is $2
print arg 3 is $3
50
Positional Parameters
Here is an example of the use of the newpos script:
$ newpos 76 trombones
starting args are 76 trombones
number of args is 2
arg 1 is 76
arg 2 is trombones
new args are NCC 1701 Enterprise
number of args is 3
arg 1 is NCC
arg 2 is 1701
arg 3 is Enterprise
51
Positional Parameters
•
What if there are more than 10 positional parameters?
$ cat ten_args
print arg 10 is $10
$ ten_args a b c d e f g h i j
arg 10 is a0
•
Use the shift command to perform a shift of the positional
parameters n positions to the left.
(Default n=1)
•
The shell variables $# and $* also change when the shift
command is used (along with the values of the positional
parameters.
Once a default shift is performed, the first parameter is
discarded unless it is saved in a user-defined variable
•
52
Positional Parameters
Examples of shift usage:
$ cat shift_it
print $#: $0: $*
shift
print $#: $0: $*
shift
print $#: $0: $*
shift 3
print $#: $0 $*
$ shift_it It was the best of times, it
12: shift_it: It was the best of times,
11: shift_it: was the best of times, it
10: shift_it: the best of times, it was
7: shift_it: times, it was the worst of
was the worst of times.
it was the worst of times
was the worst of times
the worst of times
times
$ cat ten_args
arg1=$1
shift
print $arg1 $*
$ ten_args 1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
53
Referencing Variables
•
•
So far we have seen that the syntax for referencing a
variable (obtaining its contents) is $var_name
An alternate syntax which has more capabilities is
${var_name} This syntax:
•
 Allows a script to easily use more than 10 positional parameters
 Allows for more general use of a variable contents
 Allows for the number of characters in the variable’s string
contents to be determined
 Allows for parameter substitution: replacement of the value of a
variable with an different value based on specific conditions and
events.
54
Referencing Variables
 Greater than 10 positional parameters:
$ cat moreargs
#!/bin/ksh
print arg 10 is
print arg 10 is
print arg 11 is
print arg 11 is
$10
${10}
$11
${11}
$ moreargs what rough beast, its time come at last,
slouches toward Bethlehem
arg 10 is what0
arg 10 is toward
arg 11 is what1
arg 11 is Bethlehem
55
Referencing Variables
 More general use
$ cat copy
set -x
file=lone
cp $file $filewolf
cp $file ${file}wolf
ls lone*
$ copy
+ file=lone
+ cp lone
Insufficient arguments
Usage: cp [-aDfirRp] [-b size] [-tP -[e size]] f1 f2
cp [-aDfirRp] [-b size] [-tP -[e size]] f1 ... f2 dir
+ cp lone lonewolf
+ ls lone lonewolf
lone
lonewolf
 Length of a Variable’s contents
$ cat length
book="Pillars of the Earth"
print length of book= ${#book}
$ length
length of book= 20
56
Parameter Substitution
•
Misnamed: should be variable substitution
•
Four syntaxes for parameter substitution
${var_name: - value}
${var_name: = value}
${var_name: + value}
${var_name: ? value}
•
The meanings and typical usages of these four methods are
shown on the following pages
57
Parameter Substitution
${parameter:-value}
•
•
•
•
If parameter exists and isn't null, return its present
contents; otherwise return value.
Purpose: returning a default value if the variable is
undefined.
If parameter is null, the value returned is temporary. The
null value of the variable is not permanently altered
Interactive Examples:
$ count=""
$ print You have ${count:-5} cards
You have 5 cards
$ print $count
$ FOO=/usr/local/bin/xx
$ print edit file ${FOO:-$HOME/xx}
edit file /usr/local/bin/xx
$ print $FOO
/usr/local/bin/xx
58
Parameter Substitution
•
•
As the name implies, this particular syntax of parameter
substitution can be used positional parameters.
Consider the following script that searches for a file:
$ cat fnddemo
#!/bin/ksh
set -x
find ${2:-$HOME} -name $1 -print
$ fnddemo bk.txt.Z .
+ find . -name bk.txt.Z -print
./bk.txt.Z
$ fnddemo dale11.ps
+ find /homea/dje -name dale11.ps -print
/homea/dje/dale11.ps
59
Parameter Substitution
${parameter:=value}
•
•
•
•
If parameter exists and isn't null, return its present
contents; otherwise set it to value and then return value
Purpose: Setting a variable to a default value if it is
undefined.
This form of parameter substitution cannot be used with
positional parameters because their values can only be
changed with the set command
Interactive Example:
$ count=""
$ print You have ${count:=5} cards
You have 5 cards
$ print $count
5
60
Parameter Substitution
${parameter:+value}
•
•
•
•
This substitution format acts in an opposite manner to the
previous two.
If parameter exists and is not null, return value; otherwise
return null.
Purpose: Testing for the existence of a variable
Interactive Example:
$ cat trace
set -x
trace=$1; print ${trace:+"Trace mode on"}
$ trace y
+ trace=y
+ print Trace mode on
Trace mode on
$ trace2
+ trace=
+ print
$
61
Parameter Substitution
${parameter:?message}
•
•
•
•
If parameter exists and isn't null, return its contents;
otherwise print
scriptname: parameter: message
and abort the current script. Omitting message produces
the default system message.
Purpose: Catching errors that result from variables being
undefined which need to be.
Examples:
$ cat fnd2
find ${2:-$HOME} -name ${1:?"No file specified"}
$ fnd2
fnd2: 1: No file specified
$ cat fnd3
find ${2:-$HOME} -name ${1:?} -print
$ fnd3
fnd3: 1: parameter null or not set
-print
62
Parameter Substitution
•
NOTE: In all forms, the colon (:) is optional. If omitted, the
operator tests for existence only, i.e., change "exists and
isn't null" to "exists" in each definition.
63
Advanced Pattern Matching
•
Korn shell enhancements to the set of pattern-matching
operators:
Operator
Meaning
${variable#pattern}
If the pattern matches the beginning of the
variable's value, delete the shortest part that
matches and return the rest.
${variable##pattern} If the pattern matches the beginning of the
variable's value, delete the longest part that
matches and return the rest.
${variable%pattern} If the pattern matches the end of the variable's
value, delete the shortest part that matches and
return the rest.
${variable%%pattern} If the pattern matches the end of the variable's
value, delete the longest part that matches and
return the rest.
Mnemonic: # matches the front because number signs precede numbers;
% matches the rear because percent signs follow numbers.
64
Advanced Pattern Matching
•
Typical use of the above pattern-matching operators is
stripping off components of pathnames such as directory
prefixes and filename suffixes.
•
Example: Assume the variable TEST has the value
/home/job/src/long.file.name then:
Expression
${TEST##/*/}
${TEST#/*/}
${TEST}
${TEST%.*}
${TEST%%.*}
Result
long.file.name
job/src/long.file.name
/home/job/src/long.file.name
/home/job/src/long.file
/home/job/src/long
65
Exercises
1. Write a script called lsc which executes the command ls -C. To
execute this command you must give the full path name to your lsc
script. Make the lsc shell script executable and run it.
2.Write a script called b which changes your current directory to /bin,
displays the current directory, and then executes the lsc script
created above. Make the b script executable and run it. What is your
current directory when the b script is finished executing? Why?
3.Write a script called ex_on, which turn on execute permission on the
first argument on the command line.
4.Modify ex_on to turn on execute permission on all arguments on the
command line.
5.Write a script called 12 that prints the twelfth argument on the
command line.
66
Exercises
6. Suppose you keep a file of your home coin collection that keeps track
of how many coins you have in a given category. Lines in this file
look like:
62
11
36
9
U.S.
U.S.
U.S.
U.S.
proofs
pennies (1850-1908)
pennies (1909-1950)
nickels (1861-1938)
Write a script that prints the N types of coins of which you have the
most. The default for N is 10. The program should take one argument
for the name of the input file and an optional argument for how many
lines to print.
67
Decision Making Commands
•
•
•
Flow Control
Exit Status II
if/then/else Command
•
•
•
•
•
•
Condition Tests
Logical Operators
Negation Operator
Illustrative Examples
Case Statement
Exercises
68
Flow Control
•
Flow control gives a programmer the power to specify that
only certain blocks of shell commands are executed
•
Korn shell supports the following decision making constructs:
if/else
Execute a list of statements if a certain condition is/is
case
not true
Execute one of several lists of statements depending
on the value of a variable
69
Exit Status II
•
•
For decision making, we will look at the exit status in a
different way then we did before
The constructs if, while, until and the logical AND (&&)
and OR (||) operators use exit status to make logical
decisions:
A 0 value is a logical "true" (success)
A nonzero is a logical "false" (failure)
•
There are built-in true and false commands which you can
use. The true command returns an exit status of 0, the false
command returns an exit status of 1.
70
Exit Status of a Script
•
A script, like any other process, sets an exit status when it
finishes executing. Shell scripts will finish in one of the
following ways:
Abort - If the script aborts due to an internal error, the exit status is
that of the last command (the one that aborted the script).
End - If the script runs to completion, the exit status is that of the last
command in the script
Exit - If the script encounters and exit command, the exit status is
that set by that command
• Syntax for the exit command is exit [num]. When the exit
command is encountered the script ends right there and
the exit status is set to num
71
if / else
•
Simplest type of decision making construct is the conditional
embodied in Korn shell's if statement.
•
Syntax:
if condition
then
statements
[elif condition
<-- can use multiple elif clauses
then statements . . .]
[else
statements]
fi
<-- must be by itself on the final
line of the construct
72
if / else
•
The if statement uses an exit status to determine whether or
not to execute the commands.
•
Statements are executed only if the given condition is true.
•
If one or more elifs are used, the last else clause is an "if all
else fails" part.
73
Condition Tests
•
The if construct can only test exit status but that doesn't limit
you to checking only whether commands ran properly or not.
•
Using the [[ ]] operator, many different attributes can be
tested:
– string comparisons
– file attributes
– arithmetic conditions
• [[ condition ]] just returns an exit status that tells whether
condition is true or not
• [[ ]] surround expressions that include various types of
relational operators.
74
Condition Tests: Strings
String comparison operators:
Relational Operator
str = pat
str != pat
str1 < str2
str1 > str2
-n str
-z str
True if ...
str matches pat.
str does not match pat.
str1 is less than str2.*
str1 is greater than str2.*
str is not null (has length greater than 0).
str is null (has length 0).
*based on the ASCII value of their characters
str refers to an expression with a string value, and pat
refers to a pattern that can contain wildcards.
75
String Comparison Examples
$ cat compstr
x=castle; y=cast*
if [[ $x = $y ]]
then
print $x matches $y
else
print $x does not match $y
fi
$ compstr
castle matches cast*
$ cat compstr1
x=hello; y=helper
if [[ $x > $y ]]
then
print "$x is > $y"
else
print "$x is not > $y"
fi
$ compstr1
hello is not > helper
76
String Comparison Examples
$ cat compstr2
if [[ -n $1 ]]
then
print First argument is $1
fi
if [[ -z $1 ]]
then
print Where is the first argument?
fi
$ compstr2 Logan
First argument is Logan
$ compstr2
Where is the first argument?
77
Condition Tests: Files
File attribute checking:
Relational Operator
-a file
-d file
-f file
-r file
-s file
-w file
-x file
-O file
-G file
file1 -nt file2
file1 -ot file2
True if ...
file exists
file is a directory
file is a regular file
there is read permission on file
file is not empty
you have write permission on file
you have execute permission on file, or
directory search permission if it is a directory
you own file
your group id is same as file's
file1 is newer than file2
file1 is older than file2
There are 21 such operators in total.
78
File Checking Demo Directory
•
Consider a directory that contains the following files and
subdirectories
$ls -lrt
-rwxr--r--rw-r--r--rw-r--r-drwxrwsr-x
-rw-rw-r-drwxr-sr-x
drwxr-sr-x
-rw-r--r--rw-r--r--rw-r--r--rwxr-xr-x
-rwxr-xr-x
-rw-r--r--
1
1
1
2
1
2
2
1
1
1
1
1
1
dje
dje
dje
dje
dje
dje
dje
dje
dje
dje
dje
dje
dje
appl
appl
appl
appl
appl
appl
appl
appl
appl
appl
appl
appl
appl
5
5
5
512
5
512
512
0
0
0
80
80
0
Jun
Jun
Jun
Jul
Sep
May
May
May
May
May
May
May
May
27
27
27
3
4
18
18
18
18
18
18
18
18
2000
2000
2000
2000
2000
11:54
11:54
11:54
11:55
11:57
12:01
12:01
12:04
file
my_prog
my_scr.1
scripts
slide1.ps
probs
solutions
court.msg
heel
greed
file.c
file.f90
typescript
79
File Checking Examples
$ cat fexist
if [[ -a $1 ]] then
print File: $1 does exist
else
print File: $1 does not exist
fi
$ fexist sort.c
File: sort.c does not exist
$ cat ffull
if [[ -s $1 ]] then
print $1 has something in it
else
print $1 is empty
fi
$ ffull heel
heel is empty
$ cat fx
if [[ -x $1 ]] then
print I can execute $1
else
print I cannot execute $1
fi
$ fx file
I can execute file
80
Condition Tests: Integers
•
Need to use a new test operator for integer comparisons
(( … )) instead of [[ … ]]
Relational Operator
<
>
<=
>=
==
!=
Meaning
less than
greater than
less than or equal
greater than or equal
equal
not equal
81
Integer Comparison Examples
$ cat intcomp
if (( $1 < $2 )) then
print $1 is less than $2
fi
if (( $1 <= $2 )) then
print $1 is less than or equal to $2
fi
if (( $1 == $2 )) then
print $1 is equal to $2
fi
if (( $1 > $2 )) then
print $1 is greater than $2
fi
if (( $1 >= $2 )) then
print $1 is greater than or equal to $2
fi
if (( $1 != $2 )) then
print $1 not equal to $2
fi
82
Integer Comparison Examples
$
3
3
3
intcomp 3 89
is less than 89
is less than or equal to 89
not equal to 89
$ intcomp 13 13
13 is less than or equal to 13
13 is equal to 13
13 is greater than or equal to 13
83
Logical Operators
•
Two special symbols allow the user to combine relational tests
into more complex conditions
&& is a logical AND
|| is a logical OR
•
How they work:
statement1 && statement2
statement2 will execute only if statement1 was successful
(returned an exit status of 0)
statement1 || statement2
statement2 will execute only if statement1 was not
successful (returned a nonzero exit status).
84
Logical Operators
•
The logical operators && and || are commonly used with
if/else commands
•
Examples:
$ ls -l
-r-xr-xr-x
1 dje
appl
365 May 18
-rw-r--r-1 dje
appl
209 May 18
$ cat setw
#!/bin/ksh
if [[ -f $1 && -x $1 ]] then
print You are getting write permission for $1
chmod u+w $1
fi
$ setw length.txt
$ setw intcomp
You are getting write permission for intcomp
$ ls -l
-rwxr-xr-x
1 dje
appl
365 May 18
-rw-r--r-1 dje
appl
209 May 18
15:13 intcomp
15:12 length.txt
15:13 intcomp
15:12 length.txt
85
Logical Operators
$ cat gone
#!/bin/ksh
if [[ $1 = quit || $1 = bye ]] then
print You have closed correctly
else
print Improper closing word
exit 11
fi
$ gone bye
You have closed correctly
$ echo $?
0
$ gone quit
You have closed correctly
$ echo $?
0
$ gone end
Improper closing word
$ echo $?
11
86
The Negation Operator
•
•
•
If an exclamation point ! is put before any of the conditions
discussed, it reverses the “truth or falseness” of the
condition.
[[ ! -d probs ]] returns true (exit status 0) if probs is not a
directory
[[ ! Bob = ?o? ]] returns false (exit status 1) even though
Bob matches the pattern
87
Combining Different Types of Conditions
•
If you want to combine an integer comparison with file or
string comparison within the same condition, you must do two
things:
– Use [[ … ]] for the test operator
– Use the integer relational operators shown below
Relational Operator
-lt
-le
-eq
-ge
-gt
-ne
Comparison
less than
less than or equal
equal
greater than or equal
greater than
not equal
88
Combining Conditionals Example
$ cat fileint
#!/bin/ksh
let num=$2
if [[ $1 < tact || num -ge 3 ]]; then
print Pippin
else
print Merry
fi
$ fileint ace 45
Pippin
$ fileint vice 2
Merry
89
Illustrative Examples
Test operator run interactively
$
$
0
$
$
1
[[ -z "" ]]
print $?
[[ -z foo ]]
print $?
Treating integers strings as strings: compressed format
$ if [[ 6 > 57 ]] ; then print huh?;fi
huh?
90
Illustrative Examples
The following script sets user execute permission on an
ordinary, file that is not a directory:
if [[ -f $1 && ! -d $1 ]]; then
chmod u+x $1
fi
91
Illustrative Examples
The following script removes the first file if it's older than the
second file and the variable KEY is non-null:
if [[ $1 -ot $2 && -n $KEY ]]
then
/bin/rm $1
fi
92
Illustrative Examples
The following script compares two files and if no differences are
found, removes the second file. Also helpful USAGE
programming style:
USAGE="Usage:\t $0 file1 file2"
if [[ $# -ne 2 ]]; then
print ${USAGE}
exit 1
fi
diff $1 $2 > /dev/null
if [[ $? -eq 0 ]]; then
/bin/rm $2
print $2 removed, $1 kept
else
print $1 and $2 differ
fi
93
Case command
•
•
•
•
Provides a multiple choice decision structure. When
executed one of several statement blocks will be run.
Replacement for the “elif-ladder”
Lets you test strings against patterns that can contain
wildcard characters.
Syntax:
case expression in
pattern1)
statements ;;
pattern2)
statements ;;
...
esac
94
Case Operation
•
If expression matches one of the patterns, its corresponding
statements are executed
•
If there are several patterns separated by pipes, the
expression can match any of them in order for the
associated statements to be run ( | acts as logical or )
•
Patterns
•
When the double colons (;;) are encountered at the end of
the one of the statement lists, the case terminates
are checked in order for a match; if none is found,
nothing happens
95
Case Examples
Here's a simple script which moves C and Fortran source files to
one directory and object code files to another:
case $file in
*.c|*.f)
/bin/mv $file ${HOME}/src ;;
*.o
)
/bin/mv $file ${HOME}/obj;;
*
)
print $file not moved ;; (Default value or USAGE)
esac
(Could have also used *.[cf] wildcard construct above)
96
Case Examples
•
•
The case statement is often used to specify options for a
shell script.
Here is a shell script called dt_fmat that allows the user to
enter options that affect the way the date is displayed:
case $1 in
-d) print -n Date:
date +"%a %h %d" ;;
-t) print -n Time:
date +"%T" ;;
-w) print -n Weekday:
date +"%a" ;;
-y) print -n Year: 20
date +"%y" ;;
-m) print -n Month:
date +"%h" ;;
*) print -n Date:
date +"%a %h %d"
print -n Time:
date +"%T" ;;
esac
97
Case Usage
$ date
Sat May 19 12:01:57 EDT 2001
$ dt_fmt -d
Date:Sat May 19
$ dt_fmt -t
Time:12:02:05
$ dt_fmt -w
Weekday:Sat
$ dt_fmt -y
Year: 2001
$ dt_fmt -m
Month:May
$ dt_fmt
Date:Sat May 19
Time:12:02:54
98
“Elif-Ladder” vs Case
•
The following script shows and “elif-ladder” and a case
statement that do the exact same thing. Note how the case
statement has a simpler syntax
if [[ $1 = cat || $1 = dog ]]; then
print You have a normal pet
elif [[ -z $1 ]]; then
print You have no pets
else
print You have a wierd pet
fi
case $1 in
cat|dog) print You have a normal pet ;;
'') print You have no pets ;;
*) print You have a wierd pet ;;
esac
•
Advantage of “elif-ladder”: not limited to string comparisons
99
Exercises
1.Write a script called char that checks a single character on the
command line, c. If the character is a digit, digit is displayed. If the
character is an upper or lowercase alphabetic character, letter is
displayed. Otherwise, other is displayed. Have the script print an
error message if the argument c is more than one character in length.
2.Write a script called mycopy that copies a source file to a directory.
Add a check to see if the source file exists. If the source file does not
exist, print an error message.
3.Write a script called mydir that prints the message File is a directory if
the file is a directory.
100
Exercises
4.Write a script called ver that accepts one argument on the command
line indicating the name of a file. The file is copied to another file with
the same name with the addition of the extension .v2. Also, the line
#Version 2 is added to the top of the file.
5.Execute the ver script on itself, creating a new version of the file
called ver.v2.
6.Rewrite ver.v2 to accept a possible second argument. If two
arguments are entered, the file specified by the first argument is
copied to a file with the name of the second argument. If no second
argument is entered, the file is copied to another file with the
same name, adding the extension .v2. In either case, the line
#Version2 is added to the top of the file.
101
Looping Commands
•
•
•
•
•
for
select
while
until
Exercises
102
for Loop Command
•
•
•
The for loop allows a section of commands to be run a fixed
number of times
During each time through an iteration, the loop variable is set
to a different value.
Syntax:
for name [in list]
do
statements that can use $name
done
where
— name is the loop variable which can be called anything
— list is a list of strings (defaults to $*) separated by spaces
• Operation: name is set to each string in list, in order, for each
iteration; the number of iterations equals the number of names
in list.
103
for Loop Examples
Outputting a Phrase
$ cat simple
for word in This is a test
do
print $word
done
$ simple
This
is
a
test
<-- Omitting the in This is a test phrase
and instead running the script as
$ simple This is a test
would yield the same output.
104
for Loop Examples
The list can be the contents of a local variable:
Check to see who is logged on the machines listed in the variable
SYSTEMS="myrtle gull sandy newport daytona":
for sys in $SYSTEMS
do
finger @$sys
print
done
105
for Loop Examples
list can contain shell wildcards and command substitution
as well:
$ cat file1
for i in *
do
print $i
done
$ file1
file1
file2
$ cat file2
for i in $(ls)
do
print $i
done
$ file2
file1
file2
106
for Loop Examples
•
Can explicitly make the list all the command line arguments
(also notice the alternate syntax):
$ cat upper
#!/bin/ksh
for bug in $* ; do
typeset -u day=$bug
print $day
done
$ upper Robson Bond Green
ROBSON
BOND
GREEN
107
select Command
• select allows you to generate simple interactive ASCII menus
easily.
•
Syntax:
select name [in list]
do
statements that can use $name
done
•
This is the same syntax as the for loop except for the keyword
select. As with for, in list defaults to $* if omitted.
108
select Operation
•
Each item in list is displayed as the choice in the menu with a
number preceding it. At the bottom of the menu the PS3
prompt is displayed and script is waiting for the user to select
a choice by entering its number. (A default value for the PS3
prompt is provided by the system but the user can change it)
•
The selected choice is stored in name and the selected
number in a system-predefined variable called REPLY
•
The statements in the body of the do loop are run
•
Repeats the process forever; exit loop with break statement
(or user can issue ctrl-d)
109
select Example
The following script termselect allows you to select a terminal
setting:
PS3='terminal? '
select term in vt100 vt220 xterm
do
if [[ -n $term ]]; then
TERM=$term
print TERM is $term
else
print invalid choice
fi
done
110
select Example
$ termselect
1) vt100
2) vt220
3) xterm
terminal? 4
invalid choice
terminal? 3
TERM is xterm
111
select Example
$ cat travel
select i in Past Present Future; do
print You have picked choice $REPLY
print "You will go to the $i"
break
done
$ travel
1) Past
2) Present
3) Future
#? 1
You have picked choice 1
You will go to the Past
$ travel
1) Past
2) Present
3) Future
#? 3
You have picked choice 3
You will go to the Future
$
112
while Command
•
•
Allows a section of code to be run repetitively while a certain
condition holds true.
Syntax:
while condition
do
statements ...
done
•
Operation: while condition remains true (exit status 0) the
body of the while loop is run
• [[ ]]operator can be used here as with if to specify the
condition
•
•
Beware of creating an infinite loop (condition must become
false at some point).
If condition is false when a while loop is first encountered, the
loop will not be executed even once
113
while Example
Example: print out positional parameters:
$ cat args
while [[ $# -ne 0 ]]; do
print $1
shift
done
$ args There are 7 “deadly sins”
There
are
7
deadly sins
114
while Example
Example: Using a while as a counting loop:
$ cat whileint
print $#: $3
let i=1
while (( i <= 5 )); do
shift
let i=i+1
done
print $#: $3
$ whileint 1 3 5 7 9 11 13 15 17 19
10: 5
5: 15
115
until Command
•
•
Allows a section of code to be run repetitively until a certain
condition becomes true.
Syntax:
until condition
do
statements ...
done
•
Operation: while condition remains false (exit status nonzero)
the body of the until loop is run
• [[ ]]operator can be used here as with if to specify the
condition
•
•
Typically, condition is false when an until loop is first
encountered; thus the loop will not be executed at least once
Just about any until can be converted to a while by simply
negating the condition
116
until Example
•
Classic example: checking user input
$ cat prompt
#!/bin/ksh
until [[ $ans == no ||
print -n 'Continue?
read ans
done
$ prompt
Continue? (yes or no):
Continue? (yes or no):
Continue? (yes or no):
Continue? (yes or no):
$ prompt
Continue? (yes or no):
Continue? (yes or no):
Continue? (yes or no):
Continue? (yes or no):
$
$ans == yes ]]; do
(yes or no): '
y
sure
ok
yes
n
nope
no way
no
117
Exercises
1.Write a script called lis that uses a for loop to display all files and
directories in the current directory.
2. Write a script called dirs that displays only the names of the directory
files in your current directory.
3. Using the select command, write a script that creates a menu of the
abbreviations for all the months of the year. If a certain month
abbreviation is chosen, the script should print out the full name of the
month and and number of days in it. Change the PS3 prompt to
something appropriate.
4. Using a while loop, write a script that displays a guessing game. The
player is to guess a number between 0 and 59. If the guess is too low,
the message “Higher” is printed. If the guess is too high the message
“Lower” is printed. Guesses continues in until the player guesses the
correct number. The number the user is trying to guess is taken from
the seconds of the date command.
118
Exercises
5. Modify the script from problem 4 to keep track of the number of
guesses it takes for the user to discover the correct number. Have the
script print the total number of guesses that it took.
6. Rewrite the first while example script shown in this chapter (printing
out the positional parameters) using the until statement instead.
119
Working with Integer Variables
•
•
•
•
•
Arithmetic Operations
Operator Precedence
Numerical Bases
String Variables vs Integer Variables
Exercises
120
Arithmetic Operations
•
•
•
•
So far we have seen how to give values to integer variables and
compare them (in two different ways!)
In this section, integers will be used in arithmetic operations
Below is a list of the most common operations performed with
integers
Arithmetic Operator
+
-
addition
*
multiplication
/
division with truncation
modulus division
%
•
•
Meaning
subtraction
The shell interprets symbols surrounded by $(( and )) as
arithmetic expressions.
Variables in arithmetic expressions do not need to be preceded by
dollar signs.
•
121
Arithmetic Examples
Demonstration of the integer divisions:
$ cat intop
let x=31
let y=8
print The integer division
print The modulus division
$ intop
The integer division of 31
The modulus division of 31
of $x by $y is $((x/y))
of $x by $y is $((x%y))
by 8 is 3
by 8 is 7
Parentheses can give operations higher precedence:
$ cat paren
#!/bin/ksh
let x=$(( $1 + $2 * $3 - $4 ))
print $x
let x=$(( ($1 + $2) * ($3 - $4) ))
print $x
$ paren 2 4 6 3
23
18
122
Different Numerical Bases
Possible to represent integers in alternate bases:
$ cat base
#!/bin/ksh
print -n $((
print -n ,
print -n $((
print -n ,
print -n $((
print -n ,
print -n $((
print
$ base
57,57,57,57
$
57 ))
16#39))
8#071))
2#00111001))
123
String Variables vs Integer Variables
Arithmetic does not work with string variables:
$ cat intstr
print As integers ...
if (( $1 < $2 )) then
print $1 is less than $2
fi
let x=$2+4; print $2+4 is $x
print As strings ...
if [[ $1 > $2 ]] then
print $1 is greater than $2
fi
x=$2+4; print $2+4 is $x
$ intstr 3 15
As integers ...
3 is less than 15
15+4 is 19
As strings ...
3 is greater than 15
15+4 is 15+4
124
Exercises
1. Write a script which finds the sum of all the positional parameters.
2. Write a script to determine if the first positional parameter is prime.
3. Write a script to perform the factorial of the first positional parameter
4. Write a script that will output positive and negative integers.
5. Write a script that will convert the first positional parameter from base
10 to base 5.
6. Write a script that reads in a time in seconds and outputs as
hours:minutes:sec.
125
Script Input / Output
•
•
•
•
•
•
Input / Output Redirection
“Here-is” documents
print
read
Quoting
Exercises
126
Input / Output Redirection
•
The basic I/O redirectors >, < and | are good enough for most
tasks.
•
The Korn shell supports 16 redirectors in total.
•
Here are some other useful ones:
Redirector
Function
>> file
>| file
Direct stdout to file, appending to file if it already
exists
force stdout to file even if noclobber is set
<< label
n> file
n<file
>&n
<&n
“here-is” document (see below)
direct file descriptor n to file
set file as file descriptor n
duplicate stdout to file descriptor n
duplicate stdin from file descriptor n
127
Input / Output Redirection
Example:
This script called start which can be used to start a different
script (entered as positional parameters) in the background
and save both stdout and stderr in a single log file.
$* > logfile 2>&1 &
If we want to do the same as above but to the screen as well:
$* 2>&1 | tee logfile &
The command tee takes its stdin and copies it to stdout and to
the file given as argument.
128
“Here-is” documents
•
Special form of inline input redirection
•
Syntax:
command << label
input for command
....
label
•
The shell performs command and variable substitution
where appropriate unless label is surrounded by quotes
129
print
•
We have already seen many examples of the print command.
•
Here are some useful escape sequences and options; enclose
escape sequences in quotes (either double or single):
Escape Sequence
\a
\b
\r
\t
Character printed
bell
backspace
return
tab
Option
-n
-r
-s
-un
Function
omit the final newline
ignore any escape sequence
print to history file
print to file descriptor n
130
print Example
$ cat escape
#!/bin/ksh
print "Call me \t Ishmael"
print -r "Call me \t Ishmael"
$ escape
Call me
Ishmael
Call me \t Ishmael
$ cat print.u3
#!/bin/ksh
print -u3 "My errors:" 3>errors
$ ls errors
Cannot access errors: No such file or directory
$ print.u3
$ cat errors
My errors:
131
read
•
We have seen simple examples of read.
•
Shell scripts with read are useful for certain tasks such as
reading in lines from a file
•
Suppose we have the file /etc/nicks :
myrtle
killian
daytona
...
beach_bum
brewster
speedster
which is a list of hostnames followed by nicknames. We want
to assign the variable NICKNAME to the appropriate nickname
depending on which machine you log into.
132
read
•
The following code will do it:
machine=$(uname -n)
while read host nick; do
if [[ $host = $machine ]]; then
NICKNAME=$nick
print NICKNAME is $NICKNAME
break
fi
done < /etc/nicks
<-- read exits with status 1 when there is
nothing left to read
133
Quoting
•
•
•
•
A source of confusion for novice and experienced Korn
shell script writers is when to use single quotes ‘ ‘ and
when to use double quotes “ “
Throughout the workshop we have learned that certain
symbols have special meanings: $name indicates the
contents of the variable name, $(command) indicates the
output of a command, etc.
Single quotes are strong quotes: they prevent the special
meaning symbols from performing their tasks. The symbols
are treated just like normal characters.
Double quotes are weak quotes: they allow the following
actions to take place:
– variable and positional parameter substitution
– command substitution
– arithmetic calculations
134
Quoting Demonstration
$ cat quote
#!/bin/ksh
money=franc
print "Nonsense words ${money} $1 $(date +%T) $((3*75))"
print 'Nonsense words ${money} $1 $(date +%T) $((3*75))'
$ quote raven
Nonsense words franc raven 16:23:30 225
Nonsense words ${money} $1 $(date +%T) $((3*75))
135
Exercises
1.Write a shell script tfile which creates a test file called junk. Do not use a cp
or mv command to accomplish this. File junk contains the following:
This is a test
abcdefghijklmnopqrstuvwxyz
0123456789
end of test
2. Create three files called data0, data1, and data2 which contain the month, day,
and year respectively of your birthday. Write a single script that reads from all
three of these files and then outputs your birthday to the screen
136
Communicating with Running Scripts
•
•
Sending Signals
Signal Handling
137
Signal Handling
•
We want to have control over our shell scripts even when they
are interrupted by signals.
•
Signals come in the form of control keys, the kill command
and user-initiated events like logging out.
•
Programs can be set up to "trap" specific signals and process
them in their own way.
•
Syntax of trap command:
trap command sig1 sig2 ...
when any of sig1, sig2, etc., are received, run command,
then resume execution
•
138
Signal Handling
•
After command finishes, the script resumes execution just
after the command that was interrupted.
• sig1, sig2, ... can be given by name or number (name is
more portable).
• trap without arguments lists traps that have been set
139
Signal Handling
Simple example - shell script loop :
trap 'print You hit ctrl-c!' INT
while true: do
sleep 60
done
ctrl-c will not kill this program but rather print the message and
start another sleep; type ctrl-\ to stop it (then rm core).
140
Signal Handling
Enhancing the script to handle background execution (trap the kill
command):
trap 'print
trap 'print
while true:
sleep
done
You hit ctrl-c!' INT
No kill I!' TERM
do
60
if run as loop &, kill %loop won't stop it; need kill -KILL %loop
Note: kill -l will give the numbers associated with given signals
(defined in
/usr/include/sys/signal.h).
141
Signal Handling
Examples:
• If any of the signals listed (0, 1, 2, 3, or 15) are received
by the program, file temp$$ will automatically be removed.
Without the exit command, program execution would
continue from the point where the signal was received:
trap 'rm -f temp$$; exit 0' 0 1 2 3 15
•
The command listed for trap is null. This specifies that the
signals listed are to be ignored, and the program will try to
continue where it left off:
trap "" 0 1 2 3 15
•
Using the null command (:) specifies that the signals listed
are to be ignored for the next command only:
trap : 0 1 2 3
142
Signal Handling
•
What if you want a program running in the background to
continue even after you log out?
Shell sends the HUP (hangup) signal to all of your
background jobs when you log out.
•
Use the command nohup to put a job into the background
which will ignore HUP and QUIT (ctrl-\) signals:
nohup $* > logfile 2>&1 &
143
Signal Handling
Most common signals:
Signal
1
2
3
9
15
24
Name
HUP
INT
QUIT
KILL
TERM
TSTP
Note
Hangup
ctrl-c
ctrl-\
Cannot be trapped
default kill signal
ctrl-z
144
Script Functions
•
•
•
•
•
•
Why Use Functions?
Syntax & Use
Auto-Loading
Function Definition Location
Command Precedence
Exercises
145
Functions
•
Improves shell's programmability
– already in memory (unless autoloaded)
– modular programing
•
Syntax:
function functname {
shell commands
}
or,
functname ()
{
shell commands
}
146
Functions
•
Delete a function definition with
$ unset -f functname
•
Display all defined functions:
$ functions
147
Functions
•
Two important differences between functions and shell
scripts run by name (Method 2):
– functions do not run in a subprocess; behave more like a script run
by Method 1
– functions have precedence over scripts
•
Where to put them?
– enter on command line (no longer available if you log out and log
back in)
– .profile (only available in login shell)
– $ENV file (available to all child shells)
– in $FPATH directory (Korn shell will use autoload feature)
•
•
Function names should begin with a letter or underscore
The closing brace (}) must appear on a line by itself
148
Functions
Example: Run a program and view the output
$
>
>
>
>
>
>
>
myrun ()
{
cd $WORKDIR
a.out < data.in
more data.out
/bin/rm data.in
mv data.out data.in
}
149
Exercises
1.Write a script which sets the PAGER environment variable. The script
should accept input values from:
• command-line arguments:
• -p pager or -ppager
• shell environment variable PAGER
• default setting: more -s
Include a usage line. (Hint: You should try using the pattern matching
operators introduced in this section)
2. Write a script that implements a simple version of the shell's built-in
whence command. It should check each of the directories on your
PATH for a command given as an argument, and print the full
pathname if the command is found. Only files that are executable by
you (and are not directories!) should be returned.
150
Exercises
3. Design functions pushd and popd which implement a "stack" of
directories (as in the C shell).
• The stack is just a list of directories, stored in an environment
variable we'll call DIRSTACK.
• pushd should "push" a specified directory onto the stack, i.e.,
prepend the directory name to DIRSTACK.
• popd "pops" a directory off the stack, i.e., cds to the top directory on
the stack and then removes that directory from the stack.
• Your functions should:
– Allow the use of "." to specify the current directory.
– Return an error message if pushd is invoked without an argument, or if the
specified directory does not exist.
– popd should return an appropriate message and do nothing if the stack is
empty.
•
•
Why are these implemented as functions rather than scripts?
What modifications (if any) should you make to your ~/.profile in
order to use them?
151
Exercises
4. Write a script that will change the suffix of all the files in the present
directory from .old to .new
152
HPC Designed Scripts
•
•
•
•
•
•
•
Repeating Program Execution
Debugging Options
Batch Jobs
Performance Tool Preprocessing
Multiple Tool Runs
Multiple Tool Reports
File Renaming
153
Repeating Program Execution
•
•
It is often desired to run a program several times. Even a serial programs
may have to run repeatedly to debug intermittent problems, find logical
errors, etc.
In HPC parallel processing arena, there are even more reasons to write a
script that will execute code many times:
–
Each run is with a different number of processors so that the traditional speedup
graph data can be collected (and compared with Ahmdal’s Law)
–
To perform repeated
•
•
•
–
–
•
timing of the entire or partial code.
Several parallel machines do not allow dedicated use of processors
A severe load imbalance may come and go depending on how concurrently the various
processors actually work
Over synchronized program’s times can vary due to the order of arrival of the processors at
barriers
To gather data from program output for statistical analysis. One can also gather
timing data for similar reasons.
To confirm that a parallel program is preventing race conditions which can arise
when one processor is interrupted by another when the former is trying to read,
update, and write back to a shared memory location
Repeated runs is easily done in our introductory triangle script by using a
while loop. Here the script will run the timearea code which times how
long the program takes to calculate the area
154
Timing Script
$ cat triangle
cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c
cat << EOF > sides.in
8 3 7
EOF
let i=0
while (( i < $2 )); do
$1 < sides.in >> $1.dat
let i=i+1
done
cat $1.dat
rm $1 sides.in
155
Timing Script Usage & Output
$ triangle timearea 17
1.215377
1.227504
1.197454
1.193690
1.220033
1.190733
1.194406
1.221519
1.201336
1.187614
1.197026
1.220640
1.217389
1.218919
1.192417
1.218107
1.228289
156
Debugging Options
•
To use debugging software (dbx, Totalview, Xbrowse, etc)
programs need to be compiled with certain options.
•
The debugging option is traditionally –g but has been improved
lately to include various levels of optimizations while the
debugging is being performed (G0,G1,…)
•
For all debugging options, the compiler will make a symbol table
which connects a program identifier (variable name, etc.) with a
memory address. This allows the debugger to track the values of
any variables as the program is executed line by line
•
On next page we show a variation of the introductory triangle
script in which a decision is made- through the use of a second
positional parameter- if debugging will be implemented
157
Debugging Script
$ cat triangle
if [[ $2 = debug ]]; then
cc -g -o $1 $1.c
print "$1 ready to be debugged"
else
cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1
$1.c
$1 < sides.in > $1.dat
cat $1.dat
rm $1
fi
$ triangle area
Area of Triangle is 10.392305
$ triangle area debug
area ready to be debugged
158
Batch Jobs
•
•
•
•
There are a number of popular batch processing software
that are heavily used on HPC machines: NQE, PBS, and
LSF to name three.
At this point in the course, we now realize that the “batch
files” we are used to writing are actually just shell scripts.
The header lines have the syntax and are treated as
comments if the batch script is not submitted to the batch
software.
But since these batch scripts have designed purpose – to
run the UNIX commands in them and send a log file backwe can tailor them to reflect it.
For example, it is recommended that the shell command
echoing feature be turned on so that the commands appear
in the log file preceding their output. An example is shown
on the next slides.
159
A Batch Script
•
Here is a simple batch job (file sp_gravity.job):
#QSUB -r grav
#QSUB –lT 3000
#QSUB –lM 10Mw
#QSUB -oe
set -x
# echo commands
cd gravity/batch
ls -lrt
f77 -o spg sp_gravity.f
./spg < region.rsp
ls –lrt
160
A Log file (grav.o3613)
+ ls -lrt
total 1512
-rw-r----1 dje
G-541
46098 May 18 12:22 sp_gravity.f
-rw-r----1 dje
G-541
637236 May 18 12:22 newbedrx.cray
-rwxr----1 dje
G-541
299 May 18 12:31 region.prm
-rw-r--r-1 dje
G-541
16 May 18 12:31 region.rsp
-rw-r--r-1 dje
G-541
125 May 18 12:55 sp_gravity.job
+ f77 -o spg sp_gravity.f
sp_gravity.f:
MAIN xy2glqr:
"sp_gravity.f", line 346: Warning: local variable "zero" never used
gauleg:
+ ./spg
+ 0< region.rsp
Parameter file:
Gravity output file:
region.out
3-D ORTHOGONAL COLLOCATION ON FINITE ELEMENTS **********
(GAUSS-LEGENDRE QUADRATURE)
...
Integration over source region in progress . . .
161
Log file (cont.)
+ ls -lrt
total 1712
-rw-r-----rw-r-----rwxr-----rw-r--r--rw-r--r--rwxr-xr-x
-rw-r--r--rw-r--r--
1
1
1
1
1
1
1
1
dje
dje
dje
dje
dje
dje
dje
dje
G-541
G-541
G-541
G-541
G-541
G-541
G-541
G-541
46098
637236
299
16
125
67228
2128
14502
May
May
May
May
May
May
May
May
18
18
18
18
18
18
18
18
12:22
12:22
12:31
12:31
12:55
12:58
12:58
13:00
sp_gravity.f
newbedrx.cray
region.prm
region.rsp
sp_gravity.job
spg
region.coeff
region.out
162
qsub Command with options
•
•
The realization that a batch job is really just a normal shell
script, is clear when the script only consists of the Unix
commands.
The “header” section is replaced by options to the qsub
command:
qsub –r grav –lT 3000 –lM 10Mw –eo < grav.ksh
•
Where grav.ksh is the remaining Unix commands:
set -x
cd gravity/batch
ls -lrt
f77 -o spg sp_gravity.f
./spg < region.rsp
ls –lrt
•
This is the alternative ( not recommended) method for
running a batch job. It will return the same log file shown
on the previous slides.
163
Advanced Batch Scripting
•
Since a batch script is – just that – a script, we can using
any part of the scripting language in it.
•
Consider our “classic” compile and run script we have
been using in the previous slides. If the compilation is a
failure, there is no point in proceeding with the script’s
commands since there will be no executable to work with.
•
In the advanced batch script shown on the next slide we
use the if/then shell command to check to see if there were
no errors produced in the compilation. If there was, we exit
the script (and the job,BTW).
•
The log file returned after submitting this script is also
shown. An error was deliberately inserted in the source
code.
164
Error-Checking Batch Script
set -x
cd SV1/dje/gravity
f90 -o spg sp_gravity.f
if [[ $? -ne 0 ]]; then
exit 33
fi
./spg < region.rsp
165
Error-Checking Log File
+ cd SV1/dje/gravity
+ f90 -o spg sp_gravity.f
do i=1,
^
f90-197 f90: ERROR XY2GLQR, File = sp_gravity.f, Line
= 408, Column = 14
Unexpected syntax: "operand" was expected but found
"EOS".
Cray Fortran: Version 3.5.0.4 (u15f51p42138g41109a18)
Cray Fortran: Mon Jun 10, 2002 17:13:34
Cray Fortran: Compile time: 0.3487 seconds
Cray Fortran: 1130 source lines
Cray Fortran: 1 errors, 0 warnings, 0 other messages,
0 ansi
Cray Fortran: "explain f90-message number" gives more
information about each message
+ [[ 1 -ne 0 ]]
+ exit 33
166
Performance Tool Preprocessing
•
•
•
•
Almost all of the sophisticated software tools that exist on
HPC machines to profile your code and aid in serial, vector,
and parallel optimization require the user to run the code
and generate a raw data file. The tool subsequently
analyzes the raw data under the user control (usually
interactively)
For some HPC software tools the time to gather the raw
data can be long, especially if the code is complex with
many functions and subroutines. Or the syntax required
might be different than ever other optimization tool on that
machine.
Because of these reasons, it is often useful to perform the
generation of raw data as a shell script.
The script shown on the next slide, generates the raw data
file for the prof tool on the Cray SV1.
167
Using the prof preprocessing script
oscb:$ cat profraw.ksh
#! /bin/ksh
f90 –G1 –o $1 –lprof $1.f
./$1 < $1.in
# Creates the prof.data file
prof –x –m prof.data $1 > $1.prof.raw
# User can pick the raw output filename
rm $1
oscb:$ profraw.ksh poisson
oscb:$ profview poisson.prof.raw
168
Multiple Tool Runs
•
•
•
•
For HPC tools that access Hardware Counters on the actual
vectorizing processors or commodity Microprocessors
(Pentiums, Itantiums, Ultra-Sparcs, MIPS R12K, etc.) there are too
many counter chip to gather all the information with just one run
of an optimization tool.
Therefore, to access all the hardware use information, your code
must be run several times. For Cray machines, 4 runs of hpm and
perf are required. For Microprocessor-based MPP machines
there a typically 64 counters only two of which can be read out in
a single run of the appropriate tool (ssrun for the Origin 2K,
collect for the Sun Fire 6800, etc.)
Compiling and running an executable, with different options
chosen to access various counters is an ideal task for shell
programming.
In the next slide, a script is shown that gathers all the hardware
use info for Cray machines with the hpm and perf tools installed
169
total_hw.ksh
#! /bin/ksh
# First parameter is the source code “prefix”
cc –O4 -TARG:madd=on -o $1 $1.c
hpm –g0 ./$1 > $1.hpm
hpm –g1 ./$1 >> $1.hpm
hpm –g2 ./$1 >> $1.hpm
hpm –g3 ./$1 >> $1.hpm
# Recompile with perf options
cc –F –lperf –o $1 –O4 –TARG:madd=on $1.c
typeset –x PERF_GROUP=0 PERF_DATA=p0.raw ./$1
typeset –x PERF_GROUP=1 PERF_DATA=p1.raw ./$1
typeset –x PERF_GROUP=2 PERF_DATA=p2.raw ./$1
typeset –x PERF_GROUP=3 PERF_DATA=p3.raw ./$1
cat p0.raw p1.raw p2.raw p3.raw >perf.data
rm $1
170
Saving Multiple Tool Reports
•
•
•
•
A powerful feature of HPC Optimization software tools is
that they provide information on a variety of performance
topics with just one run of the tool. A certain tool might
have a report of subroutine time-profiling, a caller-callee
tree, a summary report, etc.
When interactively using such a tool, a user can pick which
report to read, and then pick another and so on.
It is often desirable to just save a select set of the possible
reports. This can be done by running the tool several times
but with a different option each time. The option determines
what report will be saved in that run. Again, a natural job for
a shell script.
In the next slide, a script is shown that saves several
reports from the flowview tool used on Cray PVP
machines. (12 reports are available from this tool)
171
flowview report selection script
oscb:$ cat flowchoice.ksh
#! /bin/ksh
# Script assumes raw data file flow.data exists
# First parameter is prefix of the source code
pre=$1
shift
for opt in $x
do
flowview –L$opt >> pre.flow
done
oscb$ flowchoice.ksh neuralnet m h u k
172
Renaming Files
•
•
•
•
Specifically, renaming the prefixes or suffixes of a large
number of files. Many HPC parallel programs will produce
hundreds of files which contain data at certain time steps,
or spatial extent. Data visualization files are one example of
this type of output.
Say the user needs to put a new suffix or prefix on a large
set of files in order that they not be overwritten by a
subsequent data set.
After this course, one’s mind should naturally think “ I bet a
script could do this for me”. You would win that bet.
On the next slide is shown a script that will change the
suffix of all files whose name has the syntax strain001.txt,
strain002.txt to strain500.txt to the same prefix but with
“txt” replaced by “old”. The script can easily be made more
general purpose as well.
173
Renaming script
oscb$ cat suffix.ksh
#set –x
for file in $*
do
mv $file ${file%.*}.old
done
oscb$ suffix.ksh strain*.txt
174