Working
under your
qandc
account
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page
1
Connection from PC to qandc server
• In order to get into the qandc servee, we need a secure shell
application that facilitates the connection.
• One simple software that is free and it is also downloaded in
our computers at CSUSM computer labs is called putty.
• PuTTY is a free implementation of Telnet and SSH (secure
shell) for Windows and Unix platforms, along with an Xterm
terminal that allows us to interactively with Unix operating
system.
• You can download putty to your home computer. The site is:
http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page
2
Downloading Putty
• Once you enter the web site, click on the following link that
shows below:
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page
3
Downloading Putty
• Save the application anywhere in your computer. I usually put it in
my desktop.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page
4
Installing Putty
• Now double click on the downloaded putty application to get
connected to the qandc server
• Once you double click you should see the following. Click on the
run button to start the application.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page
5
Connection to qandc server using Putty
• Make sure the category is session. The port is 22, and connection
type is SSH. Then type qandc.csusm.edu for the Host Name.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page
6
Connection to qandc server using Putty
• Type your user id and password and you enter the UNIX
environment
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page
7
Connection to qandc server using Putty
• Type “ls” at the prompt line to see the content of the current
directory.
• You may not have any file under your home directory at this time.
• In mine, my files and folders are shown in different colors.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page
8
Connection to qandc server using Putty
• You can remove the default colors setting, as follows. Right click
on the header of the terminal and select “Change Setting…” from
the menu.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page
9
• The following window
appears.
• Select “Colours” from
the Category section as
shown below and
uncheck the “Allow
terminal to specify ANSI
colours” option.
• Then click on “Apply”
button to save the setting
• This gives simple black
and whiter terminal.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 10
Logging out from Unix
• We will discuss the UNIX commands later.
• For now, just type “logout” to exit from Unix and close the
terminal.
• In conclusion, putty is very simple to install and work with;
however, it is limited to simple connection to a particular system
such as qandc or cocco or bioinfo servers.
• We all want to be able to download our files and place them under
our qandc account and. We need to be able to transfer files into
and from qandc server.
• Thus, we need to have a secure shell that has the capability of
transferring files as well.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 11
Windows SSH Secure Shell Initial Connection
and Transferring Files.
• First we need to download this application. For now, you can
download this under your H drive to test.. Then you can download
it on your PC at home.
• Go to:
http://www.colorado.edu/its/docs/authenticate/printouts/win_ssh.html
• Click on the link that is shown below:
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 12
Installing SSH Secure Shell
• Save the application. You can place it under the H drive.
• Double-click on the application and press “run” button to start
installation.
• Click “next to continue:
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 13
Installing SSH Secure Shell
• Accept the License agreement, then click browse and under path
type H:\SSH. This installs the application under your H drive .
• Next follow the rest of the download procedure until it is
complete.
• You should be able to see the following icon, which is “SSH
Secure File Transfer Client” under your H drive.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 14
Running SSH Secure Shell
• To run the program and getting connected to the qandc server
simply double click on the application
• The following window opens.
• Click on “Quick connect” to start the connection to qandc server
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 15
Connecting to qandc server using SSH
• Type “qandc.csusm.edu” for the Host Name and your username
for the User Name section. Make sure the port is 22.
• Click “Connect” and it should ask you for the password.
• Note that since this is our private server, the password is
permanent and we do not require our users to change it every few
months. It is up to you what to set as your password and when you
prefer to change it.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 16
Connecting to qandc server using SSH
• Once you are successfully connected, you should see the content
of your PC on the left side of the window and the content of your
qandc server on the right side of the window, as shown below
• The same way you navigate through your folders in your PC, you
can navigate through your folders in your qandc account
Your PC
directories
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
your qandc
server
directories
California State University San Marcos (CSUSM)
Page 17
Navigating through your directories
• Click on the “home” icon of your PC section and the “home” icon
of your qandc section.
• The PC section shows your drives. You can go to any folder or any
drive in your PC.
• As you see the default directory of your qandc account is
/home/userid
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 18
File Transfer between your PC and your qandc
• Transferring a file between the your PC and your qandc account is
as simple of drag and drop
• Look for a text file of your choice under any of your drives on
your PC section and drag and drop it to your /home/username
folder of your qandc.
• For example, I have chosen the file “StrainB39_Plot1.xls” file and
drag it into the “/home/ahadaegh/ folder of my qandc account
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 19
File Transfer between your PC and your qandc
• Note that the selected file that I dragged into my qandc account is
now successfully copied and it shown under my home folder of
my qandc account
• Further, the status of all transferred files are also shown in the
status window section of the SSH application
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 20
File Transfer between your PC and your qandc
• Similarly you can take any file from your qandc account and
transfer it to any folder in your PC.
• You can also transfer the whole directory from/to your PC or your
qandc account.
• When you transfer the whole directory, files under that directory
are queued and they get transferred one by one.
• Select a directory on your PC and drag and drop it to your qandc
server. Make sure the directory is not very big in size; otherwise, it
may take a while before all the files are transferred.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 21
File Transfer between your PC and your qandc
• You can discover about all the available buttons in the SSH
application yourself . For example, one of the button that may
become handy is the hide/show folder button.
• Click the “hide/show local folder” button of your PC section and
your qandc section. An explore folder structure appears which
allows you to navigate through the folders a lot simpler.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 22
File Transfer between your PC and your qandc
• Another option that is quite handy is the “refresh” button.
• Occasionally, when you transfer files from/to your PC or qandc
account you may not see the file immediately.
• What you can do is to click on the refresh button to refresh the
directory content.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 23
Creating Terminal
and
working with UNIX
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 24
Opening a session
• You can click on the “New Terminal Window” icon to open a
session.
• Now you can use Unix commands to communicate with the
system interactively.
• In this section we introduce simple Unix commands and show
how each command correspond to the ones in window.
This is
your
terminal
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 25
What is UNIX?
• UNIX is an operating system which was first developed in the 1960s,
and has been under constant development ever since.
• By operating system, we mean the suite of programs which make the
computer work.
• It is a stable, multi-user, multi-tasking system for servers, desktops and
laptops.
• UNIX systems also have a graphical user interface (GUI) similar to
Microsoft Windows which provides an easy to use environment.
• However, knowledge of UNIX is required for operations which aren't
covered by a graphical program, or for when there is no windows
interface available
• There are many different versions of UNIX, although they share
common similarities. The most popular varieties of UNIX are Sun
Solaris, GNU/Linux, and MacOS X.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 26
The UNIX operating system
• The UNIX operating system is made up of three parts; the kernel, the
shell and the programs.
The kernel
• The kernel of UNIX is the hub of the operating system: it allocates
time and memory to programs and handles the file store and
communications in response to system calls.
• As an illustration of the way that the shell and the kernel work
together, suppose a user types rm myfile (which has the effect of
removing the file myfile).
• The shell searches the filestore for the file containing the program
rm, and then requests the kernel, through system calls, to execute
the program rm on myfile.
• When the process rm myfile has finished running, the shell then
returns the UNIX prompt [ahadaegh@qandc ~]$ to the user,
indicating that it is waiting for further commands
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 27
The shell
• The shell acts as an interface between the user and the kernel.
• When a user logs in, the login program checks the username and password, and
then starts another program called the shell.
• The shell is a command line interpreter (CLI). It interprets the commands the
user types in and arranges for them to be carried out. The commands are
themselves programs: when they terminate, the shell gives the user another
prompt ([ahadaegh@qandc ~]$ on our systems).
• The adept user can customize his/her own shell, and users can use different
shells on the same machine. The shell that we use is called bash.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 28
Files and processes
• Everything in UNIX is either a file or a process.
• A process is an executing program identified by a unique PID
(process identifier).
• A file is a collection of data. They are created by users using text
editors, running compilers etc.
• Examples of files:
• a document (report, essay etc.)
• the text of a program written in some high-level programming language
instructions comprehensible directly to the machine and incomprehensible
to a casual user, for example, a collection of binary digits (an executable or
binary file);
• a directory, containing information about its contents, which may be a
mixture of other directories (subdirectories) and ordinary files.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 29
• All the files are grouped together in the directory structure.
• The file-system is arranged in a hierarchical structure, like an inverted tree. The
top of the hierarchy is traditionally called root (written as a slash /)
/
home
ahadaegh
CPP
Bioinfo
DataFiles
bread
NewEhux(June3,201
0)
work
dgarcia
public_htm
l
ESTWebByMohammd(April23)
Documen
t
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
SQLTables
bin
……
cdeleon
e
tcrouch
…… me.cpp
……
usr
…… …… shizer
Strain1516_Plot1.xls
……
HIV_Protease
all_databases.sq
l
CodeForFeb18.txt
California State University San Marcos (CSUSM)
QuickSearchCode.txt
Page 30
Listing files and directories
• ls (list)
• When you first login, your current working directory is your home
directory.
• Your home directory has the same name as your user-name, for
example, jsmith, and it is where your personal files and
subdirectories are saved.
• To find out what is in your home directory, type
$ ls
• The ls command ( lowercase L and lowercase S ) lists the contents
of your current working directory.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 31
Making Directories
• mkdir (make directory)
• We will now make a subdirectory in your home directory to hold
the files you will be creating and using in the course of this
tutorial.
• To make a subdirectory called unixstuff in your current working
directory type:
$ mkdir unixstuff
• To see the directory you have just created, type:
$ ls
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 32
cd (change directory)
• The command cd directory means change the current working
directory to 'directory'.
• The current working directory may be thought of as the directory
you are in, i.e. your current position in the file-system tree.
• To change to the directory you have just made, type
$ cd unixstuff
• Type ls to see the contents (which should be empty)
• Now create a subdirectory called “backup” under the unixstuff
directory
$ mkdir backup
• To see the directory you have just created, type
$ ls
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 33
Exercise 1
• On your PC create 6 simple text files called home1.txt
home2.txt, unix1.txt, unix2.txt, backup1.txt, backup2.txt
• Use SSH to transfer these files from your PC to appropriate
directories:
• Place home1.txt, and home2.txt under your home
directory, unix1.txt and unix2.txt under your unixstuff
directory and backup1.txt and backup2.txt under your
backup directory
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 34
The directories . and ..
• From you terminal Type
$ cd /home/unixstuff
to get back to your unixstuff directory, and then type
$ ls -a
• As you can see, in the unixstuff directory (and in all other directories), there are two
special directories called (.) and (..)
• In UNIX, (.) means the current directory, so typing
$ cd .
means stay where you are (the unixstuff directory). Note that there is a space between
cd and the dot
• This may not seem very useful at first, but using (.) as the name of the current directory
will save a lot of typing, as we shall see later.
• (..) means the parent of the current directory, so typing
$ cd ..
• will take you one directory up the hierarchy (back to your home directory).
• Note: typing cd with no argument always returns you to your home directory. This is
very useful if you are lost in the file system.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 35
Pathnames
pwd (print working directory)
• Pathnames enable you to work out where you are in relation to the
whole file-system.
• For example, to find out the absolute pathname of your homedirectory, type cd to get back to your home-directory and then
type
$ pwd
• The full pathname for the backup directory looks something like
this:
/home/ahmadtest/unixstuff/backup
• which means that backup is in the sub-directory unixstuff which
in turn is located under the ahadaegh sub-directory, which is
under the home sub-directory, which is in the top-level root
directory called “/”.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 36
/
home
dgarcia
bread
work
ahmadTest
……
cdeleone
tcrouch
unixstuff
home1.txt
home2.txt
backup
unix1.txt
unix1.txt
backup1.txt
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
bin
……
usr
…… ……
shizer
backup2.txt
California State University San Marcos (CSUSM)
Page 37
Understanding Pathnames
• Move to your home directory by typing
$ cd
• Now type:
$ ls unixstuff
to list the consents of your unixstuff directory.
• Now type:
$ ls backups
• You will get a message like this –
backups: No such file or directory
• The reason is, backups is not in your current working directory.
• To use a command on a file (or directory) not in the current
working directory (the directory you are currently in), you must
either cd to the correct directory, or specify its full pathname.
• To list the contents of your backups directory, you must type
$ ls unixstuff/backups
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 38
~ (your home directory)
• Home directories can also be referred to by the tilde ~ character. It
can be used to specify paths starting at your home directory. So
typing:
$ ls ~/unixstuff
• will list the contents of your unixstuff directory, no matter where
you currently are in the file system.
• What do you think the following would list?
$ ls ~
• What do you think the following would list?
$ ls ~/..
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 39
Summary
Command
Meaning
ls
list files and directories
ls -a
list all files and directories
mkdir
make a directory
cd directory
change to named directory
cd
change to home-directory
cd ~
change to home-directory
cd ..
change to parent directory
pwd
display the path of the current
directory
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 40
Exercise 2
•
•
•
•
Go to the site http://genome.jgi-psf.org/
Under Amoebozoa, choose Dictyostelium purpureum QSDP1
Click on the DOWNLOAD to get into the download page
Click on OK to accept the Policy
• Download the following files and place them under your desktop
• Proteins: Dicpu1_best_proteins.fasta.gz
• Transcripts: Dicpu1_best_transcripts.fasta.gz
• Genes: Dicpu1_best_genes.gff.gz
• EST cluster consensi: Dicpu1_EST_cluster_consensi.fasta.gz
• Use either command line or SSH to Create 4 subdirectories called EST,
Protein, Gene, and Transcripts under your home directory
• Use SSH to transfer protein file into protein directory, Gene file into the
gene directory and so on
• Create a terminal (If you do not have one opened yet) and use the
commands cd, ls and pwd to explore the file system. (Remember, if you get
lost, type cd by itself to return to your home-directory)
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 41
Copying Files
cp (copy)
• cp file1 file2 is the command which makes a copy of file1 in the
current working directory and calls it file2
• To test this first, under your home directory create a sub directory
called testForCopyCmd.
• Then type:
$ cd testForCopyCmd
• Then at the UNIX prompt, type,
$ cp /home/ahmadtest/unixstuff/unix1.txt .
• Note: Don't forget the dot . at the end
• Remember, in UNIX, the dot means the current directory.
• The above command means copy the file unix1.txt to the current
directory, keeping the name the same.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 42
Moving files
mv (move)
• mv file1 file2 moves (or renames) file1 to file2
• To test this, first create couple of simple text files called p1.txt and p2.txt and
transfer them to your home directory using SSH.
• To move a file from one place to another, use the mv command. This has the
effect of moving rather than copying the file, so you end up with only one file
rather than two.
• It can also be used to rename a file, by moving the file to the same directory, but
giving it a different name.
• Change the directory to your unixstuff directory. Then, under the unixstuff
directory, type
$ mv ../p1.txt .
$ mv ../p2.txt .
• Now Type $ ls and to see if it has worked.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 43
Removing files and directories
• rm (remove), rmdir (remove directory)
• To delete (remove) a file, use the rm command. As an example, we are
going to create a copy of the science.txt file then delete it.
• Inside your unixstuff directory, type
$ rm p1.txt
$ ls
• You can use the $rmdir directoryName command to remove a
directory (make sure the directory is empty first).
• Try to remove the backups directory. You will not be able to since
UNIX will not let you remove a non-empty directory.
• Also, the command $ rm * removes all the files but not the the
directories under a directory. It does not remove hidden files either
• The command $ rm –r directoryName removes a directory and all of
its subdirectories and files under that directory
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 44
clear (clear screen)
• Before you start the next section, you may like to clear the
terminal window of the previous commands so the output of the
following commands can be clearly understood.
• At the prompt, type
$ clear
• This will clear all text and leave you with the % prompt at the top
of the window.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 45
Displaying the contents of a file on the screen
• cat (concatenate)
• The command cat can be used to display the contents of a file on
the screen.
• From your desktop, create a text file called science.txt.
• Go the following site:
http://cropsci.illinois.edu/academics/grad/bioinformatics.cfm
and copy the article and paste it into the science.txt file
• Then use SSH to transfer the file under your home directory.
• Now from the terminal, go to your home directory and type
cat science.txt
• As you can see, the file is longer than the size of the window, so it
scrolls past making it unreadable
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 46
Less
• The command “less” writes the contents of a file onto the screen a
page at a time. Type:
$ less science.txt
• Press the [space-bar] if you want to see another page, and type
[q] if you want to quit reading.
• As you can see, “less” is used in preference to cat for long files.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 47
Head
• The head command writes the first ten lines of a file to the screen.
• First clear the screen then type
$ head science.txt
• Then type:
$ head -5 science.txt
• What difference did the -5 do to the head command?
tail
• The tail command writes the last ten lines of a file to the screen.
• Clear the screen and type
$ tail science.txt
• How can you view the last 15 lines of the file?
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 48
Simple searching using “less”
• Using “less”, you can search though a text file for a keyword
(pattern). For example, to search through home1.txt for the word
'science', type
$ less science.txt
• then, still in “less”, type a forward slash [/] followed by the word
to search for the word “science” as follows:
/science
• As you can see, “less” finds and highlights the keyword. Type “n”
to search for the next occurrence of the word “science”.
• To search the previous word, you can hold the shift key down and
type “n”.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 49
“grep” command for searching words
• grep is one of many standard UNIX utilities. It searches files for
specified words or patterns. First clear the screen, then type
$ grep science science.txt
• As you can see, grep has printed out each line containing the word
science.
• Try typing
$ grep Science science.txt
• The grep command is case sensitive; it distinguishes between
“Science” and “science”.
• To ignore upper/lower case distinctions, use the -i option, i.e. type:
$ grep -i science science.txt
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 50
• To search for a phrase or pattern, you must enclose it in single
quotes (the apostrophe symbol). For example to search for
spinning top, type
$ grep -i 'Crop Sciences' science.txt
• Some of the other options of grep are:
• -v display those lines that do NOT match
• -n precede each matching line with the line number
• -c print only the total count of matched lines
• Try some of them and see the different results.
• Don't forget, you can use more than one option at a time.
• For example, the number of lines without the words science or
Science is:
$ grep -ivc science science.txt
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 51
wc (word count)
• A handy little utility is the wc command, short for word count. To
do a word count on science.txt, type:
$ wc -w science.txt
• To find out how many lines the file has, type
$ wc -l science.txt
• Important Note: To find out how many occurrences of a
particular word like ‘GC’ appears in the science.txt file we do the
following:
$ grep -o 'GC' science.txt | wc –w
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 52
Summary:
Command
Meaning
cp file1 file2
mv file1 file2
copy file1 and call it file2
move or rename file1 to file2
rm file
remove a file
rmdir directory
remove a directory
cat file
display a file
less file
display a file a page at a time
head file
display the first few lines of a file
tail file
display the last few lines of a file
grep 'keyword' file search a file for keywords
wc file
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
count number of lines/words/characters in file
California State University San Marcos (CSUSM)
Page 53
Exercise 3
• Under your desktop, create a file called fasta.txt.
• Go to the site:
http://www.dalkescientific.com/writings/NBN/parsing.html
• Copy the couple of sequences that is shown there and paste it
into your fasta.txt file
• Save the file, and use SSH to place it under your home
directory
• Use grep command to search for the number of sequences.
You can do that by finding the number of lines that
contains ‘>’
• Find out the total number of lines that have at least one
GC word in there
• Find out the total number of GC words in the file
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 54
Appending files and Redirection
• Create couple of files called list1.txt and list2.txt in your desktop.
• Write the following text in list1.txt:
• Peach
• Orannge
• Grape
• Do the same for list2.txt and write
• Pear
• Bannana
• Apple
• Use SSH to transfer these two files in your home directory in unix
• The following command
$ cat list.txt list2.txt > biglist.txt
• Appends the content of the two files and put the result into another
file called biglist.txt
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 55
Sorting the data in a file
• The command sort alphabetically or numerically sorts a list. Type
• $ sort biglist.txt
• The sorted result appears on the screen.
• You can sort the data and place the result in another file as
follows:
• $ sort biglist.txt > slist.txt
• Use cat to read the contents of the file slist.txt
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 56
• What if we have multiple numerical and non-numerical data like
the following:
Sequence
Name
tRNA
------------CELF22B7
1
CELF22B7
2
CELF22B7
3
CELF22B7
4
CELF22B7
5
tRNA Bounds
Begin End
------12619 12738
19480 19561
26367 26439
26992 26920
23765 23694
tRNA
Type
---Leu
Ser
Phe
Phe
Pro
Anti
Codon
----CAA
AGA
GAA
GAA
CGG
Intron
Begin
----12657
0
0
0
0
Bounds
End
----12692
0
0
0
0
Cove
Score
----60.01
80.44
80.32
80.32
75.76
• Then the following sorts it based on the 9th column (Cove Score)
and redirect the results into a file called “sortedData.txt” as
follows:
$ sort –k9n myData.txt > sortedData
• Note that –k9 refers to the column 9 and option ‘n’ means it is a
numerical column.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 57
• To sort the same thing is revere order, we add the option ‘r’ .
$ sort –k9nr myData.txt > sortedData
• You can also sort based on multiple columns like:
$ sort –k9n –k4 > sortedData
• This command sorts it based on column 9 and then within column
9 it sorts it based on column 4.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 58
• To see who is on the system with you, type
$ who
• Also typing:
$ who | sort
• This gives the same result as “who”, but quicker and cleaner.
• To find out how many users are logged on, type
$ who | wc -l
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 59
Summary
Command
Meaning
command > file
redirect standard output to a file
command >> file
append standard output to a file
command < file
redirect standard input from a file
command1 | command2
pipe the output of command1 to
the input of command2
cat file1 file2 > file0
concatenate file1 and file2 to
file0
sort
sort data
who
list users currently logged in
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 60
Exercise 4
• Go to the following site
http://wheat.pw.usda.gov/NSF/curator/nsf_contributor/ksu/ksu_genbank_number.txt
• Copy the data (not the header part) and place into a file called
wheatEST.txt under your desktop
• Transfer the file into your EST or your home directory
• Sort the file based on the 1nd column and direct the results
into a file called “sorted-EST-Id.txt”. Note that the first
column is numeric
• Sort the file based on the 3rd column and direct the results
into a file called “sorted-AccNo.txt”.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 61
The * wildcard
• The character * is called a wildcard, and will match against none
or more character(s) in a file (or directory) name.
• For example, in your unixstuff directory, type
$ ls list*
• This will list all files in the current directory starting with the word
“list”
• Or typing”
$ ls *list
• will list all files in the current directory ending with the word
“list”
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 62
On-line Manuals
• There are on-line manuals which gives information about most commands.
• The manual pages tell you which options a particular command can take, and
how each option modifies the behavior of the command. Type man command
to read the manual page for a particular command.
• For example, to find out more about the wc (word count) command, type:
$ man wc
• Alternatively,
$ whatis wc
• gives a one-line description of the command, but omits any information about
options etc.
• When you are not sure of the exact name of a command,
$ man –k keyword
• For example, $ man -k copy prints all the command that explains about the
copy command in unix
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 63
File system security (access rights)
• In your unixstuff directory, type
$ ls -l
• Where “l” represents “long listing” . You will see that you now get
lots of details about the contents of your directory, similar to the
example below.
• Each file (and directory) has associated access rights, which may
be found by typing ls -l. Also, ls -lg gives additional information
as to which group owns the file (beng95 in the following
example):
drwxr-xr-x 3 ahadaegh faculty 4096
Feb 5 2009 BioInfo
• In the left-hand column is a 10 symbol string consisting of the
symbols d, r, w, x, -, and, occasionally, s or S.
• If d is present, it will be at the left hand end of the string, and
indicates a directory: otherwise - will be the starting symbol of the
string.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 64
• The 9 remaining symbols indicate the permissions, or access
rights, and are taken as three groups of 3.
• The left group of 3 gives the file permissions for the user that
owns the file (or directory) (ahadaegh in the above example);
• the middle group gives the permissions for the group of people to
whom the file (or directory) belongs (faculty in the above
example);
• the rightmost group gives the permissions for all others.
• The symbols r, w, etc., have slightly different meanings depending
on whether they refer to a simple file or to a directory.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 65
Access rights on files.
• r (or -), indicates read permission (or otherwise), that is, the presence or
absence of permission to read and copy the file
• w (or -), indicates write permission (or otherwise), that is, the permission
(or otherwise) to change a file
• x (or -), indicates execution permission (or otherwise), that is, the
permission to execute a file, where appropriate
• Access rights on directories
• r allows users to list files in the directory;
• w means that users may delete files from the directory or move files into it;
• x means the right to access files in the directory. This implies that you may
read files in the directory provided you have read permission on the
individual files.
• So, in order to read a file, you must have execute permission on the directory
containing that file, and hence on any directory containing that directory as a
subdirectory, and so on, up the tree.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 66
Some Example:
-rwxrwxrwx
A file that everyone can read, write and execute
(and delete).
-rw-------
A file that only the owner can read and write,
no-one else can read or write, and no-one has
execution rights (e.g. your mailbox file).
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 67
chmod (changing a file mode)
• Only the owner of a file can use “chmod” command to change the
permissions of a file. The options of “chmod” are as follows
Symbol
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
Meaning
u
user
g
group
o
other
a
all
r
read
w
write (and delete)
x
execute (and access directory)
+
add permission
-
take away permission
California State University San Marcos (CSUSM)
Page 68
• For example, to remove read write and execute permissions on the
file biglist for the group and others, type
$ chmod go-rwx biglist.txt
• This will leave the other permissions unaffected.
• To give read and write permissions on the file biglist to all,
$ chmod a+rw biglist.txt
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 69
Processes and Jobs
• A process is an executing program identified by a unique PID
(process identifier). To see information about your processes, with
their associated PID and status, type :
$ ps
• A process may be in the foreground, in the background, or be
suspended. In general the shell does not return the UNIX prompt
until the current process has finished executing.
• Some processes take a long time to run and hold up the terminal.
• Backgrounding a long process has the effect that the UNIX
prompt is returned immediately, and other tasks can be carried out
while the original process continues executing.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 70
Running background processes
• To background a process, type an & at the end of the command line. For
example, the command sleep waits a given number of seconds before
continuing. Type:
$ sleep 10
• This will wait 10 seconds before returning the command prompt
[ahadaegh@qandc ~]$. Until the command prompt is returned, you can do
nothing except wait.
• To run sleep in the background, type
$ sleep 10 &
[1] 6259
• The & runs the job in the background and returns the prompt straight away,
allowing you do run other programs while waiting for that one to finish.
• The first line in the above example is typed in by the user; the next line,
indicating job number and PID, is returned by the machine. The user is be
notified of a job number (numbered from 1) enclosed in square brackets,
together with a PID and is notified when a background process is finished.
Backgrounding is useful for jobs which will take a long time to complete.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 71
Killing a process
• It is sometimes necessary to kill a process (for example, when an
executing program is in an infinite loop)
• To kill a job running in the foreground, type ^C (control c). For
example, run
$ sleep 100
• You can type ^C too kill the process.
• You can run the program in the background as follow:
$ sleep 100 &
• Type: $ jobs to see the job number of the process:
• If it is job number 4, type
$ kill %4
• To check whether this has worked, examine the job list again to
see if the process has been removed.
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 72
ps (process status)
• Alternatively, processes can be killed by finding their process
numbers (PIDs) and using kill PID_number
$ sleep 1000 &
Now type:
$ ps
• You may getting something like this:
PID TT S TIME COMMAND
20077 pts/5 S 0:05 sleep 1000
21563 pts/5 T 0:00 netscape
21873 pts/5 S 0:25 nedit
• To kill off the process sleep 1000, type:
$ kill 20077
• Where 20077 is the process id of the job.
• Now type ps again to see if it has been removed from the list.
• If a process refuses to be killed, uses the -9 option, i.e. type
$ kill -9 20077
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 73
Summary:
Command
ls -lag
chmod [options] file
command &
^C
^Z
bg
kill %1
ps
kill 26152
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
Meaning
list access rights for all files
change access rights for
named file
run command in background
kill the job running in the
foreground
suspend the job running in
the foreground
background the suspended
job
kill job number 1
list current processes
kill process number 26152
California State University San Marcos (CSUSM)
Page 74
Other useful UNIX commands
• gzip: This reduces the size of a file, thus freeing valuable disk
space. For example, type
$ ls -l science.txt
• and note the size of the file using ls -l . Then to compress
science.txt, type:
$ gzip science.txt
• This will compress the file and place it in a file called
science.txt.gz
• To see the change in size, type ls -l again.
• To expand the file, use the gunzip command.
$ gunzip science.txt.gz
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 75
• zcat: zcat will read gzipped files without needing to uncompress
them first.
$ zcat science.txt.gz
• If the text scrolls too fast for you, pipe the output though less
$ zcat science.txt.gz | less
• file: file classifies the named files according to the type of data
they contain, for example ascii (text), pictures, compressed data,
etc.. To report on all files in your home directory, type:
$ file *
• diff: This command compares the contents of two files and
displays the differences. Suppose you have a file called file1 and
you edit some part of it and save it as file2. To see the differences
type:
$ diff file1 file2
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 76
Exercise 5
• Remember the Protein, EST, and others that you downloaded
and placed in proper directories in previous exercise,
• Use gunzip to uncompress them and use less or more
command to see the content of the file
A.R.
Hadaegh
Dr. Ahmad
R. Hadaegh
California State University San Marcos (CSUSM)
Page 77