Efficient Coding Practices
Efficient Coding Practices
OR
How I Stopped Worrying and Learned to Love ing Things!
Joshua D. Bodyfelt
Max­Planck­Institute für Physik komplexer Systeme
Condensed Matter Division
MPI­PKS Research Group Informal Seminar ­ 05.02.2010 Online at:
http://www.mpipks­dresden.mpg.de/~bodyfelt/Tutorials/Scripting.pdf
A Bit of Motivation...
A Bit of Motivation...
“Josh, you must stop working hard, and start working efficient....”
­ Moritz Hiller, after +72 hours without sleep
A Brief Synopsis
A Brief Synopsis
The Clydesdale – A Sample ForTran Code
●
The Tack – General BASH (Bourne Again Shell) Syntax
●
The Feedbag – I/O Redirection in BASH
●
The Plow – Advanced I/O in BASH: HERE Blocks
●
The Hitch­Tongue – S.G.E. Job Task Arrays
●
The Sowed Field – HERE & gracebat scripting
●
The Clydesdale – Sample Fortran Code
The Clydesdale – Sample Fortran Code
The Clydesdale – Sample Fortran Code
The Clydesdale – Sample Fortran Code
Equivalent to:
cin >> A >> w >> d >> N; (C++)
def main(A,w,d,N): (Python)
The Tack – General BASH Syntax
The Tack – General BASH Syntax
➢
Common Shell Commands
http://tldp.org/LDP/abs/html/
Informative: date hostname logname echo top ps
File­Based: basename cat cp mv rm cut chmod tar zip
Path­Based: ls pwd dirname cd (­r option on many of the above)
Operational: kill bc python FORTRAN BINARIES!!!
Variables/Arrays: Assignment & Referencing
➢
Assigned by:
scalar=value
array=( value1 value2 … )
“....” → Substitution Quotes
'.....' → Literal Quotes
Referenced as:
$ → “Substitution”
{} → Gen. (Array) Substitution
# → Array Length
[ ] → Array Index (Zero'ed)
@ → Complete Array
* → “Wild Card” (dummy)
$( ) → Command Substitution
Simple Examples
user@host> u=Koshchey; v=the; w=Deathless
user@host> r="$u $v $w"; s='$u $v $w'
user@host> echo $r; echo $s
Koshchey the Deathless
$u $v $w
user@host> t=$u $v $w
bash: the: command not found
user@host> x=(Baba Yaga\'s Chicken Leg Hut)
user@host> echo ${x}; echo ${x[@]}; echo ${#x[@]}
Baba
Baba Yaga's Chicken Leg Hut
5
user@host> echo ${x[1]}
Yaga's
user@host> ls
test test.f90 test_perm.f90
user@host> b=($(ls ­1))
user@host> echo ${b[1]}
test.f90
The Tack – General BASH Syntax, continued
The Tack – General BASH Syntax, continued
➢
Variables/Arrays: Operations
http://www.gnu.org/software/bc/manual/bc.html
String Operations:
${variable:position:characters} → Trim # of characters from the given position
${variable%pattern} → Trim the shortest match from the end
${variable%%pattern} → Trim the longest match from the end ${variable##pattern} → Trim the longest match from the beginning ${variable#pattern} → Trim the shortest match from the beginning Mathematical Operations:
$(( +­* )) → Simple Integer Arithmetic
echo “scale=... commands” | bc <­l> → Real Arithmetic
Simple Examples
String Operations
user@host> string="This is a test. It is only a test."
user@host> echo ${string:5:8}
is a tes
user@host> echo ${string%test*}
This is a test. It is only a
user@host> echo ${string%%test*}
This is a
user@host> echo ${string#*test}
. It is only a test.
user@host> echo ${string##*test}
.
Note the importance (AND POSITION) of the Wild Card!!
user@host> echo ${string##test*}
This is a test. It is only a test.
user@host> echo ${string%%test}
This is a test. It is only a test.
Mathematical Operations
user@host> a=1
user@host> b=$((a+10)); c=$((b*5))
user@host> echo $b $c
11 55
user@host> pi=$(echo "scale=9; 22/7" | bc); echo $pi
3.142857142
user@host> pi=$(echo "scale=2; 22/7" | bc); echo $pi
3.14
­l option imports a mathematics library
user@host> pi=$(echo "scale=10; 4*a(1)" | bc ­l); echo $pi
3.1415926532
The “|” character is a pipe
. It takes the stdout
stdout from one
from one
The “|” character is a pipe. It takes the command and pipes
it as stdin
stdin to the second command.
to the second command.
command and pipes it as The Tack – General BASH Syntax, continued
The Tack – General BASH Syntax, continued
➢
Flow Control
If ­ Then:
If [[ condition ]]; then
...<command list>
fi Do ­ While:
while [[ condition ]]; do
...<command list>
done Simple Examples
user@host> if [[ $animal == 'Dog' ]]; then echo $animal" is man's best friend."; fi
user@host> if [[ $((1+1)) ­gt 2 ]]; then echo "Because the Party says so."; fi
user@host> ResistCode=(Black Brown Red Orange Yellow Green Blue Violet Grey White); band=0
user@host> while [[ $band ­le $((${#ResistCode[@]}­1)) ]]; do
> echo ${ResistCode[$band]}
> band=$((band+1))
> done
Reading Parameter Files:
user@host> cat parameter.list | while read line; do echo $line; done
##### A, w, d, N #######
0.0 1.0 0.0 1000
1.0 1.0 0.0 1000
2.0 1.0 0.0 1000
3.0 1.0 0.0 1000
This barely scratches the surface, but is deep enough to start scripting. For more, check out:
http://tldp.org/LDP/abs/html/
The Feedbag – I/O Redirection in BASH
The Feedbag – I/O Redirection in BASH
(1) Pipes
We've already seen pipes, but to recap the syntax:
<command 1> | <command 2>
➔ Takes stdout of command 1 and pipes it into command 2
z.B. : Four parameters in simplewave.f90 – A (bias), w (freq), d (phz), N (samples)
echo “$A $w $d $N” | ./simplewave
(2) File Redirection
Used to write stdout to a file: command > output file
→ Overwrites existing file.
command >> output file → Appends to existing file.
Additional C­like command for redirecting stdin from file: command < input file
Often, combined as
command < input file > output file
command1 | command2 | command3 > output file
The last (and most useful) I/O redirection method gets it's own section....
The Plow – Advanced I/O in BASH: HERE Blocks
The Plow – Advanced I/O in BASH: HERE Blocks
The Problems: Suppose I want to write several lines to a file. Why should I have to write “>> file” for multiple lines? ➢
Suppose I want one script to do everything:
➢
­ Check code compilation ­ Get inputs from user or parameter lists
­ Create SGE submission scripts
­ Submit jobs
­ Run any post­processing code
­ xmgrace the output The (Very Simple) Solution:
➢ Include a HERE statement block – combo of redirections, EOF (end of file) and a 'cat'.
( cat <<EOF
…commands…
EOF
) >> output file
The command list can include anything, including variables...thus,
WARNING: Be careful with variables in HERE statement block....
WARNING:
The Plow – Advanced I/O in BASH: HERE Blocks, cont.
The Plow – Advanced I/O in BASH: HERE Blocks, cont.
BASH Script = List of Linux Terminal (“Shell”) Commands, begins with #!/<path to bash>
Simple Example – Our First Submission Script
Sharp + Bang = “Shebang”!
Whatever command follows shebang will run before the script itself. ­ Useful for 'pre­scripts'.
Loop Start: Read in parameters from file.
Alternately, can use shell math or user input (read statement).
First lines of the submission script:
shebang and grid engine setup.
S.G.E. options: #$ Typical submission script: (1) Descend to working directory.
(2) Pipe in run parameters to executable.
(3) Move (if needed) the results. Submit to the queue! TAKE NOTE OF PREVIOUS WARNING: What if we had “cd $rundir” → What would happen?
Hint: Don't forget to “chmod 755” your scripts!
The Hitch­Tongue – S.G.E. Job Task Arrays
The Hitch­Tongue – S.G.E. Job Task Arrays
Sun Grid Engine ­ Job Task Arrays:
One of the most useful SGE options is the “­t” option, which allows for 'true' distribution.
“Turn the Knob” Philosophy
Typically, we have a fixed set of parameters, and vary one. The task array allows one script to address ALL the values of the variable parameter.
(z.B. a disorder realization!).
Why would I want to use it? ­ A Typical qstat output
job­ID prior name user state submit/start at queue slots ja­task­ID
­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­
857655 0.50243 freq_run bodyfelt t 02/04/2010 21:22:30 pallas@pallas02.mpipks­dresden 1 4
857655 0.50122 freq_run bodyfelt t 02/04/2010 21:22:30 tycho@tycho28.mpipks­dresden.m 1 5
857655 0.50081 freq_run bodyfelt t 02/04/2010 21:22:30 tycho@tycho26.mpipks­dresden.m 1 6
857655 0.50061 freq_run bodyfelt t 02/04/2010 21:22:30 brahe@brahe05.mpipks­dresden.m 1 7
857655 0.50049 freq_run bodyfelt t 02/04/2010 21:22:30 merkur@merkur01.mpipks­dresden 1 8
857655 0.50041 freq_run bodyfelt t 02/04/2010 21:22:30 rhea@rhea15.mpipks­dresden.mpg 1 9
857655 0.50035 freq_run bodyfelt qw 02/04/2010 21:22:01 1 10
How it works:
(1) Set the SGE option: #$ ­t 1­<array size>
(2) At runtime, internal shell variable is created: $SGE_TASK_ID
(3) You can then use bc on $SGE_TASK_ID to get varying parameter values.... http://wiki.gridengine.info/wiki/index.php/Main_Page
The Hitch­Tongue – S.G.E. Job Task Arrays, cont.
The Hitch­Tongue – S.G.E. Job Task Arrays, cont.
Simple Example – Task Array Submission Script
....YET AGAIN.... DON'T BE STUPID! Pay attention to
$variable vs.
\$variable
The Sowed Field – HERE & gracebat scripting
The Sowed Field – HERE & gracebat scripting
http://plasma­gate.weizmann.ac.il/Grace/
Setup the axes, fonts, titles, etc.
Everything but loading data.
“Bodyfelt Custom Shop” Script:
256 x RGB colormap to grace
Loading data. Same as xmgrace's
Import ASCII > Load as Block Data
And then some. Without clicking.
Compile grace batch. Output in xmgrace.
Same idea can be applied to most command line graphics
(gnuplot, pgplot, wave, etc.)
The Sowed Field – HERE & gracebat scripting, cont.
The Sowed Field – HERE & gracebat scripting, cont.
Conclusion
Conclusion
One last word regarding scripting... One last word regarding scripting...
Повторение ­ мать учения.
➢
➢
➢
➢
➢
La práctica hace un santo.
Übung macht den Meister.
Άγιος που δε θαυματουργεί, μηδέ δοξολογιέται.