Introduction to Linux on Iceberg
•
•
•
•
What are UNIX and Linux?
The Shell
Getting Connected
Basic UNIX/Linux Commands
– Working with Directories
– Editing files
• Help!
• Running Programs
• Exercise
History of . UNIX/Linux?
• Unix operating system was developed around 1969
in the Bell Labs
• Originally written using C
• Around 1990 Linus Torvalds of Helsinki University
started off a freely available academic version of
Unix
• Linux is the Antidote to a microsoft dominated future
What is UNIX/Linux ?
• Multi-Tasking O/S
• Multi-User O/S
• Available on a range of Computers
Which UNIX?
• SunOS
Sun Microsystems
• IRIX
Silicon Graphics
• HP-UX
Hewlett Packard
• Linux
For IBM PC compatibles
There are a number of certification bodies with published
standards and test suites to ensure quality of
products. Such as;
• Posix: Portable Operating System Interface ( IEEE
standard).
UNIX Internals (Simplified)
• Kernel
– System Internals
• Shell
– Command Interpreter
– Programming language
• File System
• Process Management
UNIX Shells
•
•
•
•
•
sh Bourne Shell (Original Shell) (Steven Bourne of AT&T)
bash Bourne Again Shell (GNU Improved Bourne Shell)
csh C-Shell (C-like Syntax)(Bill Joy of Univ. of California)
ksh Korn-Shell (Bourne+some C-shell)(David Korn of AT&T)
tcsh Turbo C-Shell (More User Friendly C-Shell).
Linux and GNU
• Linux is an implementation of Unix
• Linux/Unix operating system is written in ‘C’
• Linux is not part of the GNU project but uses the same licensing
agreements
• Many of the linux utilities and tools are taken from the GNU
project.
• There are many flavours of linux distributions. The mix of the
kernel (linux) with the utilities (GNU and other) and the installation
procedure determine the flavour. Some of these are;
–
–
–
–
–
–
Fedora
SuSE
Redhat
Debian
Mandrake
Knoppix
Linux on Windows
• Shell systems within the windows environment which using the
same linux OS command structure.
• Clones of the linux OS in Windows
– Cygwin
– minimalgw
• GIT Version Control System
– GitShell
passwords
• In normal linux environment the passwd command can be used
to change the user passwords. However, because we manage
passwords centrally this command will not work on iceberg.
• If you wish to change your iceberg password you will have to do
this via a web interface at the following URL:
http://www.shef.ac.uk/cics/password
Terminal Emulation Software
Used in Sheffield Univ.
• ssh : Secure Shell Client (no XWindows)
– Example putty
– http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe
• Exceed : ( Xwindows support and also secure shell)
Starting the Putty Client.
• Host Name
– iceberg
– iceberg.shef.ac.uk
• Click on data
under the
Connection item
Setting your username with Putty.
• Select the data
item under
connection
• Enter your
username at Autologin username
– cs4mgXX
Enabling Graphical Output with Putty
Putty.
• Select the ssh
item under
connection then
select X11
• Tick the box which
says Enable X11
forwarding
• To start the client
click the open
button
Enabling Graphical Output with Putty
Putty.
• Click yes at the
ssh security
dialog box
• Enter your
password
Graphical UNIX via Exceed
•
•
•
•
X Concepts
Using the Exceed Software
Starting an xterm session
Starting a ssh session
Basic X Concepts
• X Server runs on local machine
– PC
Exceed, Cygwin, Xming
– UNIX Workstation Included in OS
– Apple Mac
Exodus
• X Client runs on remote machine
– Graphical Application
• xterm
• xcalc
• Modelling and visualisation packages etc.
ssh on Managed/XP workstations
Secure-shell client can be accessed via;
Start Unix_ConnectivityExceed3Dssh.
When the program initialises you will be asked to enter
hostname (default is iceberg) , username and password.
ssh on Managed/XP workstations
When ssh starts you will be presented with the login panel as
shown below; Default host name is iceberg.shef.ac.uk and can
be changed if you want to login somewhere else.
You may next be asked if you wish to save the public host-key
into a local-database. This is not a crucial question and you may
answer YES or NO.
Also if asked to provide host response just press ENTER.
Exceed on the stand-alone PCs
Starting an Exceed xterm session
Start Exceed ->Xstart and fill in the panel as shown below
Fill in
host,username,p/w,comm
and
and click on the run icon
to start xterm
Exceed on the stand-alone PCs
An Exceed xterm session
Accessing Iceberg Using Sun Global
Desktop.
• Currently an experimental service
• One of the easiest ways of accessing services on
Iceberg
• When a user is logged it presents menus for starting
terminals and some applications on iceberg.
Accessing Iceberg Using Sun Global
Desktop.
• Browser URL
http://myapps.shef.ac.uk
• For the first access type yes
or OK when permissions
are requested to accept
keys and run scripts.
Multiple ssh or xterm shells
There are no limits to the number of ssh or xterm windows one can start
simultaneously by methods described in the previous slides. You may
also start extra xterm windows from the host by simply typing xterm &
On iceberg we also have a local command named Xterm that starts up an
xterm window with nicer to use parameters. On iceberg we strongly
recommend that you use Qsh rather than xterm or Xterm command so
as to make use of a free worker node. Qsh will act like Xterm but will
make use of a worker node.
Typing exit will terminate an xterm or ssh session neatly.
This will also close the xterm window but not the ssh window. ssh
windows can be closed via the file exit menu.
Some basic rules
• Unix is case sensitive.
• Commands are in lower case.
• Backspace and/or Del Keys correct typing errors.
If the terminal parameters are not correctly set;
try Ctrl+H
• Ctrl+C Aborts a program or command.
• You can use the arrow keys to recall previous
commands, optionally edit and execute them.
Key combinations in bash
Ctrl+A
Move cursor to beginning of line
Ctrl+C
<Break> Stop running program
Ctrl+D
<Logout>
Ctrl+E
Move cursor to end of line
Ctrl+H
Delete character to the left
Ctrl+Z
Suspend the program
Arrow left/right
Move pointer left/right to insert chars etc.
Arrow up/down
Recall previous commands so as to edit them andsubmit
them by pressing ENTER
Tab
Command or filename completion. Auto-complete the
command line (to save typing)
If tab does not work tab again to show possibilities
Tab tab
Filenames
• Filenames can comprise of:
a-z, A-Z alphabetic characters
0-9
digits
.-_+
special characters
mon+tue_01.06-03-96
• Wildcards when referencing files
* any character or sequence of characters
? any single character
Format of Unix commands
• command [option ...] [filename ...]
eg:
ls
ls -l tutorial
more tutorial
List Directory
• ls
• ls -l
• ls -a
list directory
list directory in long format
list all (inc. hidden) files
-rw------ l course01
Access
Permissions
57
Number of
bytes in file
Oct 18 11:05
hello.c
Date and time
last modified
Directory
Structure
/ (root)
usr
cs4un1
home
cs4un2
Home directory of user cs4un1 : /home/cs4un1
When you log in you are positioned in your home directory.
The environment variable $HOME is also set to contain this
directory name.
Working with Directories
• pwd
• cd
cd
cd ..
cd mydir
cd /var/adm
print working directory
change directory
move to home directory
move up one level
move into a subdirectory
move to an absolute directory
• mkdir directory_name
• rmdir directory_name
create a new directory
delete an empty directory
Displaying contents of a text_file
•
more filename
This command will start listing the contents of filename on screen and pause after a screenfull
of data. While pausing, use the following characters to control the output.
Spacebar
next screenful
n Spacebar : next n lines
Enter
next line
b
back one screen
n b : back n screen’s full
q
quit
? or h
list commands
where n is a whole number
Displaying contents of a text_file… continued
• cat [options] filename [filename … ]
This command will output the contents of filename[s] to standard-output (
normally screen) without pausing.
Following options are useful;
-v
display non-printing characters
-n display with lines numbered on the left
• tail [-n] filename
This command lists the last 10 lines of a text file.
If a number is specified (.eg. -20 ) lists the last n (i.e 20) lines
Editors
There are a number of editors for Linux platforms most are not as
easy to use as the Windows based editors. Our recommendation
is;
Use nedit if you are using XWindows ‘e.g. Exceed’, cygwin.
Use vi or vim if you have a text terminal ‘telnet’.
•
•
•
•
nano
nedit Easy ‘graphical’ , good.
vi, vim
Require some knowledge, vim is like vi but
more and includes help
emacs
Have faithful following, good once learned.
Vi Text Editor
•
•
•
•
Very powerful
Available on all versions of UNIX
No in-built help
Man page gives detailed help
• Vi has three modes
– Command
– Text entry
– Last line
Vi Text Editor
•
•
•
•
Very powerful
Available on all versions of UNIX
No in-built help
Man page gives detailed help
• Vi has three modes
– Command
– Text entry
– Last line
Copying files
•
Copy files (optionally directories)
cp fromfile tofile
Some of the useful options are:
-R or –r : Recursive copy ‘fromfile’ is a directory so the entire
directory and its contents are copied.
e.g. cp –r mydir newdir
-p : preserve. Preserves all attributes of the file ,such as access rights and
creation date.
•
Copy and concatenate files by using cat
Cat command concatenates contents of list of files and directs the output to
standard output (normally screen). When used with redirection ‘>’ it can be
used to join files together.
e.g. cat file1 file2 file3 > new_big_file
Renaming and deleting files
•
•
mv :This command will move a file or directory to a new location. It can thus be
used to rename files/directories as well as change their locations in the global
directory structure.
Syntax:
mv source destination
Example:
mv myfile mynewfile
mv myfile subdirectory/myfile
mv mysubdir mynewsubdir
rm : This command will delete a file (optionally a directory if used with –r option).
Syntax: rm object_to_delete
Example:
rm myfile
rm –r mydirectory
Searching in files
• grep string file
This command finds and prints out the lines in the file(s) containing the specified
string
string
file
= word or phrase
= file or list of files (wild_card can be used)
Note: We strongly advise that the string is quoted.
Examples: grep ‘Green Man’ england.dat
grep ‘Zodiac’ t*.dat
grep ‘Zone[a-z]’ security.fil
Finding files and information about them
•
•
•
find :Finds a file in the directory hierarchy
find root_dir –name filename –print
Example :
find . –name “myprog.*” -print
Note that wild-character containing string must be enclosed in quotas
which : Shows in which directory a command is located.
Syntax: which command_name
file: Can be used to see what type of data a particular file contains. For example, script
, program, library, executable binary etc…
Syntax:
file command_name
Using find
• Searches recursively for specified file
– find path_list options filename(s) action
• Criteria for include
–
–
–
–
Filename, using name option
Size, using size option
Time last accessed, using atime option
Time last modified using mtime option
Find Examples
• Find a file called mystery in /bin and /usr and print
the result
– find /bin /usr –name libGL* –print
– find . –user myusername –print
• Find files accessed in the last n days
– find . –atime n –print
• Find files modified in the last n days
– find . –mtime n -print
Issuing System Commands on Files
Found
• Use the exec option
• Copy found file to a specified directory, curly braces instruct find
to substitute the name of the file in this location
–
–
–
–
find . –name “*.doc” –exec cp {} document \;
File is copied to direcory document
; is needed to terminate the execute command
\ escape character to take away special meaning of ; in find
• Reaffirm execution of system commands using –ok option
– find . -name “*.doc” -ok rm {} \;
• Search for a string in all the find files
– -find . –name “*.doc” –exec grep “Iceberg” {] \; -print
Manual Pages and Info system
Man: Manual pages : Give text-based help on usage.
Manual pages are grouped into sections (1,2,3,4 ..).
There is usually one manual page per command which is
located in one of the directories defined by the MANPATH
environment variable.
To access the Manual page for a command just type;
• man command
To get a list of manual pages that contain a ‘word’ type;
• man – k topic
Info : Similar to man but can scroll with cursor keys and link to
other information.
Running programs on iceberg
• Iceberg is the gateway to the cluster of worker nodes and the
only one where direct logging in is allowed.
• Iceberg’s main purpose is to allow access to the worker
nodes but NOT to run cpu intensive programs.
• All cpu intensive computations must be performed on the
worker nodes. This is achieved by the qsh command for the
interactive jobs and qsub command for the batch jobs.
• Once you log into iceberg, taking advantage of the power of a
worker node for interactive work is done simply by typing qsh
and working in the new shell window that is opened. This
what appears to be a trivial task has would in fact have
queried all the worker nodes for you and started a session on
the least loaded worker in the cluster.
• The next set of slides assume that you are already working
on one of the worker nodes (qsh session).
Running programs
• Two modes of operation foreground and background
– Foreground
Interact with program via
keyboard/screen
– Background
No connection with keyboard/screen
Submit to backbround by Appending ‘&’
EG:
myprog >& myfile &
The symbols ‘>&’ redirect output and any errors to the file myfile
Although the above method of running jobs on the
background is feasible, the prio we recommend that you
submit your background into the batch queue via the qsub
command.
Redirection
Most unix commands are not aware of the source of their
input or the destination of their output. They simply
read/write from/to stdin/stdout. The shell takes care of
these issues.
• Standard Input (default=>keyboard)
• Standard Output (default=>screen)
• Redirection symbols <,>,>> can be used to specify
files as the source/destination of the read/write
operations to override the above defaults.
Redirection continued …
Most unix commands are not aware of the source of their input or the
destination of their output. They simply read/write from/to
stdin/stdout. The shell takes care of these issues.
• To redirect the output to a file use the ‘>‘ symbol.
Example:
ls -l > dirlist
• The ‘>‘ symbol should be used with care as it may over-write an
existing file. ‘>>’ symbol can be used instead if the output should
be appended to the end of an existing file rather than over-writing
it.
Example:
ls -l >> logfile
• If nothing is directed to a file then a zero size file is created, or if
the file already existed then the contents of the file is removed
Example:
> afile
• The file /dev/null is a special symbol to indicate a ‘black-hole’
Example : ls –l > /dev/null
Redirection continued …
If any program expects any of it’s input from the
standard-input-channel , i.e. the keyboard, it can also
read the same information from a file by redirection.
• To read input from a file use the ‘<‘ symbol.
Example:
write cs1xyz < message.fil
Here any text input {write} program expects from the
keyboard will be simply read from a file named
message.fil.
Each end-of-line will be treated as an <ENTER> on the
keyboard.
Piping
• Feeding the output of one command into the input
of another command
• The symbol ‘|’ is called a pipe
command | command
• eg: ls -al | more
ls -la | grep Nov
Examples of re-direction and piping
•
•
•
•
•
•
•
•
•
ls –l | grep ‘Jun’
ls –l | grep `Jun` > june_files
ls –l | grep `Jun` | cut –c 57-80 > june_files
cut –c 1-10 < test_files
aspell –l < message.txt > report.txt
grep fluent < news.dat
grep fluent < news.dat | cut –c 1-72
(grep fluent < news.dat ) | cut –c 1-72
(grep fluent < news.dat ) | cut –c 1-72 > fluent.news
Foreground Program Control
• Kill a program
• Stop a program
Ctrl C
Ctrl Z
Note a stopped program still exists in the system
and hence can be re-started.
Program control within current shell
• jobs
Lists jobs (programs)
• bg %job_id
Place a job in the
background
• fg %job_id
Return a job to the foreground
• stop %job_id
Stop a job
• kill %job_id
Kill a job
Process_id can be used in place of %job_id for more
definitive way of identification.
jobs
[1] + Running
time.sh > out
stop %1
[1] + Stopped (signal) time.sh > out
bg %1
[1] + time.sh > out &
kill %1
Terminated
Program control using ‘ps’ and ‘kill’
• ps
Report process status
ps -f -u username
UID
PID PPID C
STIME TTY
username 24816 24585 0 16:23:04 pts/50
username 20169 19956 0 16:05:45 pts/50
username 24585 20169 0 16:35:07 pts/50
• kill
TIME
0:00
0:01
0:00
CMD
sleep 2
-csh
/bin/sh time.sh
Terminate process
– Find its process ID (PID) using the ps command
– Kill the process using the kill command
kill 24585
– Sometimes kill on its own does not work so try
kill -KILL 24585
or kill -9 pid
Filename Completion
• Complete a partially typed filename
– Operation
• Type enough characters to uniquely identify the name
• Press the ‘Tab’ key (for C – shell use ‘Esc’ key)
• If the response is a ‘bleep’ press Ctrl-d to list possible matches
– Setup
• Add the following to your .cshrc file
• set filec
Repeating Previous Commands
• Operation
– history
List previous commands
– !!
re-run last command
– !n
re-run the nth command
– !str
last command starting with str
eg:
!vi
• Setup
– Add the following to your .cshrc file
– set history=40
Command Aliasing
• Definitions
– alias
– alias
– alias
ll=' ls -lF '
h=' history 24 '
dir=' ls '
• Setup
– Add alias definitions to your .bashrc file
UNIX Utilities
•
•
•
•
tar
compress
sed
awk
Further Help
• Sheffield University Linux Group
– http://bit.ly/1FfW2ZX
• University of Sheffield Hpchub discussion area
– http://hpchub.sites.sheffield.ac.uk/discussion
– Or e-mail hpchub@Sheffield.ac.uk
– E-mail hpchub to join the group
• E-mail hpc@sheffield.ac.uk or helpdesk.Sheffield.ac.uk
• GNU/Linux Command-Line Tools Summary
– http://en.tldp.org/LDP/GNU-Linux-Tools-Summary/html/index.html
• Bash Guide for Beginners
– http://tldp.org/LDP/Bash-Beginners-Guide/html/index.html
• Advanced Bash-Scripting Guide
– http://en.tldp.org/LDP/abs/html/index.html
• The Linux Documentation Project
– http://en.tldp.org/