Document Sample
Unix Powered By Docstoc
					Developing Bioinformatics
   Computer Skills 1

1. What Bioinformatics Skills?

1. Operating Systems


1. UNIX/LINUX basics
Developing Bioinformatics Computer
  Developing Bioinformatics Computer Skills
Biological research using the web

• NCBI - National Center for Biotechnology Information

• ExPASy - (Expert Protein Analysis System) proteomics
  server of the Swiss Institute of Bioinformatics

• Many other examples

Methods -
Web browser                             - Interactive
API/screen scrapping                    - Scriptable
ftp                                     - Interactive/scriptable

  Developing Bioinformatics Computer Skills

• Visualizing protein structures and computing structural

• Predicting protein structure and function from sequence

• Tools for genomics and proteomics

• Automating data analysis

• Building biological databases

• Visualization and data mining

    What do bioinformaticists do? (1)

•   Computer stuff
•   Internet stuff
•   Database stuff
•   Program
•   Design user interfaces
•   Visualization

    What do bioinformaticists do ?(2)
•   Science!
•   Project management
•   Interdisciplinary communication
•   Data management
•   Data analysis
•   Data mining

What do bioinformaticists do? (3)

• Model development
• Statistical analysis
• Machine learning


1. What Bioinformatics Skills?

1. Operating Systems


1. UNIX/LINUX basics
Various Operating Systems
                          Comparison of Operation Systems
 Name        Creator      First public   Predecessor     Latest stable Latest release Cost/Availabili    Preferred     Target system
                           release                         version         date            ty            license1          type

   AIX          IBM        1986-?-?      System V R3          6.1         2007-11-?    Bundled with     Proprietary     Server,
                                                                                         hardware                       NetApp,
 FreeBSD    The FreeBSD   1993-12-?        386BSD             7.2         2009-05-04       Free            BSD           Server,
              Project                                                                                                  Workstation,
 HP-UX        Hewlett-     1983-?-?          Unix        11.31 "11i v3"   2007-02-15       $400         Proprietary     Server,
              Packard                                                                                                  Workstation
  IBM i         IBM        1988-?-?        OS/400            V6R1         2008-04-?    Bundled with     Proprietary      Server
   IRIX         SGI        1988-?-?          Unix            6.5.30       2006-08-16   Bundled with     Proprietary       Server,
                                                                                         hardware                      Workstation
GNU/Linux Linus Torvalds, 1992-08-25     Unix4, Minix5    Linux kernel 2008-12-14;         Free         GNU GPL,            See:
               et al.                                    2.6.30; GNU C 2007-10-23                       GNU LGPL       Comparison
                                                           Library 2.7                                                   of Linux
 Mac OS      Apple Inc.   1984-01-24       None2    7        9.2.2        2002-05-12   Bundled with     Proprietary       Works
                                                                                         68K and
                                                                                       Macs; versions

Mac OS X     Apple Inc    2001-03-24      NeXTStep           10.6.0       2009-08-28       $169         Open source    Workstation,
                                                                                                           core         Desktop
 NetBSD     The NetBSD     1993-05         386BSD             5.0         2009-04-29       Free            BSD           Server,
              Project                                                                                                  Workstation
Microsoft    Microsoft    1985-11-20      MS-DOS,         Windows 7                                     Propriertary    Desktop
Windows                                    OS/2,
          Beyond Using Websites
• You can do a lot of sophisticated bioinformatics using
  public websites

• But at some point you may be faced with a LOT of data

• The only solution is to have your own bioinformatics
  computer, database, and custom programs.

• Needs more processor power and more hard drive space
  than a typical desktop personal computer
Bioinformatics Requires Powerful
• One definition of bioinformatics is "the use of
  computers to analyze biological problems.”

• As biological data sets have grown larger and
  biological problems have become more complex, the
  requirements for computing power have also grown.

• Computers that can provide this power generally use
  the Unix operating system - so you must learn Unix
         Unix Runs the Internet
• Unix is a command line interface, used by most large,
  powerful computers.

• In fact, Unix is the underlying structure for most of the
  Internet and most large scale bioinformatics operations.

• A knowledge of Unix is likely to be helpful in your future
  career, regardless of where you pursue it.
Web Server Statistics

1. What Bioinformatics Skills?

1. Operating Systems


1. UNIX/LINUX basics
               Unix Advantages
• It is very popular, so it is easy to find information
  and get help
      •   pick up books at the local bookstore (or street vendor)
      •   plenty of helpful websites
      •   USENET discussions and e-mail lists
      •   most Comp. Sci. students know Unix
• Unix can run on virtually any computer
     (IBM, Sun, Compaq, Macintosh, etc)
• Unix is free or nearly free
      • Linux/open source software movement
      • RedHat, FreeBSD, MKLinux, LinuxPPC, etc.
         Stable and Efficient
• Unix is very stable - computers running Unix almost
  never crash
• Unix is very efficient
       • it gets maximum number crunching power out
         of your processor (and multiple processors)
       • it can smoothly manage extremely huge
         amounts of data
       • it can give a new life to otherwise obsolete
         Macs and PCs
• Most new bioinformatics software is created for Unix
  - its easy for the programmers
            UUNIX Philosophy

• Write programs that do one thing and do it well

• Write programs to work together

• Write programs to handle text streams, because
that is a universal interface

                          Doug Mcllroy (1978)
                             UNIX Philosophy
1. Rule of Modularity: Write simple parts connected by clean interfaces

2. Rule of Clarity: Clarity is better than cleverness

3. Rule of Composition: Design programs to be connected to other programs

4. Rule of Separation: Separate policy from mechanism; separate interfaces from engines

5. Rule of Simplicity: Design for simplicity; add complexity only where you must

6. Rule of Parsimony: Write a big program only when it is clear by demonstration that
   nothing else will do

7. Rule of Transparency: Design for visibility to make inspection and debugging easier

8. Rule of Robustness: Robustness is the child of transparency and simplicity.

            Eric, Raymond (2003) The Art of UNIX Programming
                     UNIX Philosophy (continued)
9. Rule of Representation: Fold knowledge into data so program logic can be stupid and robust

10. Rule of Least Surprise: In interface design, always do the least surprising thing

11. Rule of Silence: When a program has nothing surprising to say, it should say nothing

12. Rule of Repair: When you must fail, fail noisily and as soon as possible

13. Rule of Economy: Programmer time is expensive; conserve it in preference to machine time

14. Rule of Generation: Avoid hand-hacking; write programs to write programs when you can

15. Rule of Optimization: Prototype before polishing. Get it working before you optimize it

16. Rule of Diversity: Distrust all claims for “one true way”

17. Rule of Extensibility: Design for the future, because it will be here sooner than you think .
      Unix has some Drawbacks
• Unix computers are controlled by a command line
      • NOT user-friendly
      • difficult to learn, even more difficult to truly master

• Hackers love Unix
      • there are lots of security holes
      • most computers on the Internet run Unix , so hackers can
        apply the same tricks to many different computers

• There are many different versions of Unix with subtle (or
  not so subtle) differences
   Open Source Bioinformatics
• Almost all of the bioinformatics software that you need
  to do complex analyses is free for UNIX computers

• The Open Source software ethic is very strong among

• New algorithms generally appear first as free software
  (a publication requirement)
             Free Software
• Linux operating system, MySQL database
• Perl - programming language
• Blast and Fasta - similarity search
• Clustal - multiple alignment
• Phylip - phylogenetics
• Phred/Phrap/Consed - sequence assembly and
  SNP detection
• EMBOSS - a complete sequence analysis
  package created by the EMBL (like GCG)
  Computer Hardware is not Free
• However, you can build a powerful Linux
  server/cluster for $20-50K
     (depending on how much power you need)

• The real cost is for a person to manage the
  machines, install the software, and train scientists
  to use it.

• Small schools can join together or affiliate with a
  larger neighbor.
     Do Biologists have to become
• No, but it can give you a big advantage

• More and more of biology is becoming
  computer aided design of experiments,
  automated equipment, and computational
  analysis of the results

• “I just want to say one word to you ... Databases”
      Linux – An Operating System

• A kernel

• Surrounded by utilities and applications

• Strictly speaking, Linux is the kernel of a Unix-
  like operating system originally created by
  Linus Torvalds in 1991
•   1971 - The first edition of the Unix server operating system emerges from
    Bell Labs

•   1985 - Richard Stallman publishes his famous "GNU Manifesto", one of the
    first documents of the open-source revolution

•   1991 - In August, Torvalds announces his plans to create a free operating
    system on the Minix users newsgroup. He modestly notes in his posting that
    his OS is "just a hobby”. In October, Linux 0.01 is released on the Internet
    under a GNU public license

•   1993 – 100,000 users. Start of Slackware and Debian projects

•   1999 – 15,000,000+ users. RedHat IPO
          How Linux Came to Be –
        Intentional and Evolutionary
• "Linux, it turns out, was no intentional masterstroke,
  but an incremental process, a combination of
  experiments, ideas, and tiny scraps of code that
  gradually coalesced into an organic whole." - Glyn
  Moody, Wired

• “Once the system is at the edge of chaos, we are
  bound to see surprises. Linux is one such system that
  has come to dazzle us all. “ – Ko Kuwabara

1. What Bioinformatics Skills?

1. Operating Systems


1. UNIX/LINUX basics
             General Unix Tips
• UNIX is case sensitive!!
  – myfile.txt and MyFile.txt do not mean the same
  – I like to use capital letters for directory names - it puts
    them at the top of an alphabetical listing

• Every program is independent
  – the core operating system (known as the kernel)
    manages each program as a distinct process with its
    own little chunk of dedicated memory.
  – If one program runs into trouble, it dies, but does not
    affect the affect the kernel or the other programs
    running on the computer.
               The Unix Shell
• You communicate with a Unix computer through a
  command program known as a shell.
• The shell interprets the commands that you type on
  the keyboard.
• There are actually many different shells available for
  Unix computers, and on some systems you can
  choose the shell in which you wish to work.
• You can use shell commands to write simple
  programs (scripts) to automate many tasks
               Simple Programs
• You can use the Unix shell to run programs right from
  the command line, or save them as shell scripts

• Simple loops can run a GCG command (such as Blast or
  FASTA) on many sequence files

• Then you can check the output files for specific results,
  and use if statements to sort or take other actions
                Unix Commands
• Unix commands are short and cryptic like
  vi or rm.
      • Computer geeks like it that way; you will get used to it

• Every command has a host of modifiers which are
  generally single letters preceded by a hyphen:
      ls -l or mv -R
      • Capital letters have different functions than small letters,
        often completely unrelated.
      • A command also generally requires an argument, meaning
        some file on which it will act:
      cat -n mygene.seq
• You can substitute the * as a wildcard symbol for any
  number of characters in any filename

• If you type just * after a command, it stands for all files
  in the current directory:
               lpr *   will print all files

• You can mix the * with other characters to form a search
  ls a*.txt    will list all files that start with “a” and end in “.txt”

• The “?” wildcard stands for any single character:
  cp draft?.doc   will copy draft1.doc, draft2.doc, draftb.doc, etc.
                Typing Mistakes
• Unix is remarkably unforgiving of typing mistakes
      • You can do a lot with just a few keystrokes, but it can be
        hard or impossible to undo

• If you have not yet hit „return‟
      • The „delete‟ key removes the characters that you just typed
      • Which key on your keyboard will actually function as “delete”
        will vary depending on the type of computer that you are
        using, the Telnet program and the Unix shell that you are
        using, or if you are running a specific Unix program
            Control Characters
• You type Control characters by holding down
  the „control‟ key while also pressing the
  specified character

• While you are typing a command:
     • ctrl-W erases the previous word
     • ctrl-U erases the whole command line

• Control commands that work (almost) any time
     • ctrl-S suspends (halts) output scrolling up on your
       terminal screen
     • ctrl-Q resumes the display of output on your screen
     • ctrl-C will abort any program
            Getting Help in Unix
• Unix is not a user-friendly computer system.
   – While not actively user-hostile, it is perfectly happy to sit
     there and taunt you with a blank screen and a blinking >

• There is a rudimentary Help system which consists of
  a set of "manual” pages for every Unix command

• The man pages tell you which options a particular
  command can take, and how each option modifies
  the behavior of the command

• Type man and the name of a command to read the
  manual page for that command.
> man ls
Reformatting page. Please wait... completed

ls(1)                                                                     ls(1)


  ls - Lists and generates statistics for files


  ls [-aAbcCdfFgilLmnopqrRstux1] [file...|directory...]


  Interfaces documented on this reference page conform to industry standards
  as follows:

  ls:     XPG4, XPG4-UNIX

  Refer to the standards(5) reference page for more information about indus-
  try standards and associated tags.


  -a    Lists all entries in the directory, including the entries that begin
        with a . (dot). Entries that begin with a . are not displayed unless
        you refer to them specifically, or you specify the -a option.

  -A    [Compaq] Lists all entries, except . (dot) and .. (dot-dot). If you
        issue the ls command as the superuser, it behaves as if you specified
        this option.

  -b    [Compaq]   Displays nonprintable characters in octal notation.

  -c    Uses the time of last inode modification (file created, mode changed,
        and so on) for sorting when used with the -t option. Displays the time
        of last inode modification (instead of the time at which the file's
        contents were last modified) when used with the -l option. This option
        has effect only when used with either -t or -l or both.

manaacsba (10%)
                   More Help (?)
• The man pages, such as they are, give information
  about specific commands

• So what if you don‟t know what command you need?

• There is a command called apropos that will give you a
  list of commands that contain a given keyword in their
  man page header:
               apropos password
   – The man command with the -k modifier gives a similar result to

• Get yourself a good "Intro to Unix" book
         Unix Help on the Web
Here is a list of a few online Unix tutorials:
• Unix for Beginners

• Introduction to Unix (OSU)

• Unix Guru Universe

• Getting Started With The Unix Operating
                  Unix Filenames
• Unix is cAsE sEnsItiVe

• UNIX filenames contain only letters, numbers, and the _
  (underscore), . (dot), and - (dash) characters

• Unix does not allow two files to exist in the same
  directory with the same name.
      • Whenever a situation occurs where a file is about to be
        created or copied into a directory where another file has
        that exact same name, the new file will overwrite (and
        delete) the older file.
      • Unix will generally alert you when this is about to happen,
        but it is easy to ignore the warning.
            Filename Extensions
• Most UNIX filenames start with a lower case letter and
  end with a dot followed by one, two, or three letters:
      • However, this is just a common convention and is not
      • It is also possible to have additional dots in the filename

• The part of the name following the dot is called the

• The extension is often used to designate the type of file.
      Some Common Extensions
• By convention:
  –   files that end in .txt are text files
  –   files that end in .c are source code in the "C” language
  –   files that end in .html are HTML files for the Web
  –   Compressed files have the .zip or .gz extension

• Unix does not require these extensions (unlike
  Windows), but it is a sensible idea and one that
  you should follow
       Files and Directories:
Naming something gives you power over it.
Absolute Addressing
Addressing relative to your home dir.
Addressing relative to your current dir.
      Working with Directories
• Directories are a means of organizing your files
  on a Unix computer.
  – They are equivalent to folders on Windows and
    Macintosh computers

• Directories contain files, executable programs,
  and sub-directories

• Understanding how to use directories is crucial to
  manipulating your files on the Unix system.
       Your Home Directory
• When you login to the server, you always start
  in your Home directory

• Create sub-directories to store specific projects
  or groups of information, just as you would
  place folders in a filing cabinet

• Do not accumulate thousands of files with
  cryptic names in your Home directory
        File & Directory Commands
• This is a minimal list of Unix commands that
  you must know for file management:
 ls (list)            mkdir (make directory)

    cd (change directory)         rmdir (remove directory)

 cp (copy)                                     pwd (present working directory)

    mv (move)                                  more (view by page)

    rm (remove)                                cat (view entire file on screen)

• All of these commands can be modified with
  many options. Learn to use Unix „man‟ pages
  for more information.
• pwd (present working directory) shows the name and
  location of the directory where you are currently working:
       > pwd
                 – This is a “pathname,” the slashes indicate sub-directories
                 – The initial slash is the “root” of the whole filesytem

• ls (list) gives you a list of the files in the current directory:
       > ls
                     assembin4.fasta Misc       test2.txt
                     bin       temp      testfile

   – Use the ls -l (long) option to get more information about each file
       > ls -l
       total 1768
       drwxr-x--- 2 browns02 users 8192 Aug 28 18:26 Opioid
       -rw-r----- 1 browns02 users 6205 May 30 2000 af124329.gb_in2
       -rw-r----- 1 browns02 users 131944 May 31 2000 af151074.fasta
• cd (change directory) moves you to another
       >cd Misc
       > pwd

• mkdir (make directory) creates a new
  sub-directory inside of the current directory   >
       assembler phrap    space
       > mkdir subdir
       > ls
       assembler phrap    space   subdir

• rmdir (remove directory) deletes a sub-directory,
  but the sub-directory must be empty
       > rmdir subdir
       > ls
       assembler phrap    space
• There are some important shortcuts in
  Unix for specifying directories
     • . (dot) means "the current directory"
     • .. means "the parent directory" - the directory one
       level above the current directory, so cd .. will move
       you up one level
     • ~ (tilde) means your Home directory, so cd ~ will
       move you back to your Home.
        – Just typing a plain cd will also bring you back to your
          home directory
          Unix File Protections
• File protection (also known as permissions)
  enables the user to set up a file so that only
  specific people can read (r), write/delete (w),
  and execute (x) it

• Write and delete privilege are the same on a
  Unix system since write privilege allows
  someone to overwrite a file with a different one.
       File Owners and Groups
• Unix file permissions are defined according to
  ownership. The person who creates a file is its owner.
      • You are the owner of files in your Home directory and all its

• In addition, there is a concept known as a Group.
      • Members of a group have privileges to see each other's files.
      • We create groups as the members of a single lab - the
        students, technicians, postdocs, visitors, etc. who work for a
        given PI.
                   View File Permissions
• Use the ls -l command to see the permissions for all files in a
> ls -l
              drwxr-x--- 2 browns02 users 8192 Aug 28 18:26 Opioid
              -rw-r----- 1 browns02 users 6205 May 30 2000 af124329.gb_in2
              -rw-r----- 1 browns02 users 131944 May 31 2000 af151074.fasta

          – The username of the owner is shown in the third column. (The owner of the files
            listed above is browns02)
          – The owner belongs to the group “users”

• The access rights for these files is shown in the first column. This
  column consists of 10 characters known as the attributes of the file:
  r, w, x, and -
r indicates read permission
w indicates write (and delete) permission
x indicates execute (run) permission
- indicates no permission for that operation
  > ls -l
  drwxr-x--- 2 browns02 users 8192 Aug 28 18:26 Opioid
  -rw-r----- 1 browns02 users 6205 May 30 2000 af124329.gb_in2
  -rw-r----- 1 browns02 users 131944 May 31 2000 af151074.fasta

• The first character in the attribute string indicates if a file is a
  directory (d) or a regular file (-)

• The next 3 characters (rwx) give the file permissions for the owner
  of the file

• The middle 3 characters give the permissions for other members
  of the owner's group

• The last 3 characters give the permissions for everyone else

• The default protections assigned to new files on our system is:        -
  rw-r----- (owner=read and write,
  group =read, others=nothing)
                  Change Protections
• Only the owner of a file can change its protections

• To change the protections on a file use the chmod (change mode)
                  [Beware, this is a confusing command.]
    – First you have to decide for whom you will change the access permissions:
                   » the file owner (u)
                   » the members of your group (g)
                   » others (o) (ie. anyone with an RCR account)
    – Next you have to decide if you are adding (+), removing (-), or setting (=)

• Taken all together, it looks like this:
        > chmod u=rwx g+r o-x myfile.txt
        This will set the owner to have read, write, and execute permission;
        add the permission for the group to read;
        and remove the permission for others to execute the file named myfile.txt.
           Commands for Files
• Files are used to store information, for example,
  data or the results of some analysis.
  – You will mostly deal with text files
  – Files on the RCR Alpha are automatically backed up to tape
    every night

• cat dumps the entire contents of a file onto the
  – For a long file this can be annoying, but it can also be helpful if
    you want to copy and paste (use the buffer of your telnet
• Use the command more to view at the contents of a file
  one screen at a time:
        > more t27054_cel.pep
        !!AA_SEQUENCE 1.0
        P1;T27054 - hypothetical protein Y49E10.20 - Caenorhabditis elegans
        Length: 534 May 30, 2000 13:49 Type: P Check: 1278 ..
        t27054_cel.pep (87%)

   – Hit the spacebar to page down through the file
   – Ctrl-U moves back up a page
   – At the bottom of the screen, more shows how much of the file has
     been displayed

• More sophisticated options for viewing text files are
  available in a text editor (next week).
                       Copy & Move
• cp lets you copy a file from any directory to any other
  directory, or create a copy of a file with a new name in
  one directory
      • cp filename.ext newfilename.ext
      • cp filename.ext subdir/newname.ext
      • cp /u/jdoe01/filename.ext ./subdir/newfilename.ext

• mv allows you to move files to other directories, but it is
  also used to rename files.
   – Filename and directory syntax for mv is exactly the same as for
     the cp command.
      • mv filename.ext subdir/newfilename.ext

   – NOTE: When you use mv to move a file into another directory,
     the current file is deleted.
• Use the command rm (remove) to delete files

• There is no way to undo this command!!!
   – We have set the RCR server to ask if you really want
     to remove each file before it is deleted.
   – You must answer “Y” or else the file is not deleted.
            > ls
            af151074.gb_pr5 test.seq
            > rm test.seq
            rm: remove test.seq? y
            > ls
           Moving Files between
• You will often need to move files between
  computers - desktop to server and back

• There are several options
  –   Sneaker net (floppy, zip, writeable CD, USB)
  –   E-mail
  –   Network filesharing
  –   FTP
                FTP is Simple
• File Transfer Protocol is standard for all
  computers on any network

• The best way to move lots of data to and from
  remote machines:
  – put raw data onto the server for analysis
  – get results back to the desktop for use in papers and

• Graphical FTP applications for desktop PCs
  – On a Mac, use Fetch, RBrowserLite, Filezilla
  – On a Windows PC, use WS_FTP, Filezilla
                  FTP Login
• When you open an FTP program, you connect
  to mendel just as you would with a telnet client.
• Your username and password are the same.
• You will automatically end up in your home
• Put files from you PC to the server, Get files
  from the server to your desktop machine.
         File system commands
•   pwd - report your current directory
•   cd <to where> - change your current directory
•   ls <directory> -list contents of directory
•   cp <old file> <new file> - copy
•   mv <old file> <new file> - move (or rename)
•   rm <file> -delete a file
•   mkdir <new directory name> -make a directory
•   rmdir <directory> -remove an empty directory
          getting recursive
• remove a directory and its contents:
rm -r <directory>

• copy a directory and its contents:
cp -r <directory>
• create shortcuts for yourself
     ~>alias ll “ls -la”

• Use alias with no arguments to discover
  current aliases
      rm rm -i
      ll    ls -la

• Type “unalias rm” to remove alias.
PATH: a very important shell variable
>echo $PATH

• If a program (like ls) is in one directory found in your
  path, then typing it (~>ls <enter>) will execute it.

• Otherwise you can type the full absolute address to
  execute a program (~>/usr/bin/ls <enter>)
     finding things in your PATH.
• Type “which <command>” to find the location of the
  program which would be run when you type

• If you don‟t remember if it was chgrp or chgroup, type
  “ch<control-d>” to get a list of commands that starts
  with ch.

• when all else fails, use “find” to find a file.
~>find <start dir> -name “*.doc”
                    Shell scripts.
• If you have a bunch of commands you‟d like to
  automate, you can put them on separate lines of a file.
  Then type “source <file>” to run the script.

• If the first line of your script looks like
#!<program name>
then you can make the script executable. When it
  executes, it uses <program name> to interpret the
  contents of the script.
                 Login scripts
• Most people have a script that executes when
  they log in. It is commonly used to set up one‟s
  PATH and aliases.

• Ask someone to help you start your own login
               What not to use.

• telnet, ftp, rlogin

• all your data (including your password) is
  transmitted plain text over the network.

• from library machines you can use the java ssh
  client from a web browser.
              using ssh keys
• use “ssh-keygen” to generate a public/private
  set of keys. You keep the private key and
  append the public key to authorized_keys.

• You can now log in using either your
  password or the private key file.
      using secure copy: scp
• copy local to remote
scp <source file>

• copy remote to local
scp user@machine:<path>   <source
     pico - the pine composer
• the simplest visual editor available on
  most Unix systems.

• all possible commands displayed at
  bottom of screen. (control-somethings)

• no real surprises

• All terminal programs have:

  – standard output, which is usually your screen
  – standard input, which is usually your
  – standard error, which is also the screen
  redirect output to a file with >

• If you type who at the prompt, you will get a
  list of who is logged into the system.

• If you type who >f, a file named f will be
  created and the standard output of who will be
  placed in that file instead of to your screen.
                > vers >>

• By default, who >f will overwrite the file

• Use who >>f to append to f rather
  than overwriting it.
  redirecting input from a file
             with <
• The program sort will sort its standard input
  and then print it on standard out.

• To sort the lines of file1 and display:
sort < file1

• To sort the lines of file1 and save in file2:
sort < file1 > file2
 The output of one program
 can be the input to another.

who | sort

• The output of who is sorted and
  shown on your terminal screen.

• grep shows only those lines containing
  its search pattern.

• To see all lines in a file containing „bob‟:

grep „bob‟ < file1
          The cat command
• the arguments to cat are concatenated together
  and displayed on stdout. To view a file:

cat file1

• if no arguments, cat puts on stdout whatever you
  type on stdin, so this does the same thing:

cat < file1
     To start a process in the
      background, use “&”.

• example:

   big_program > output &

• big_program will not have input!
              managing jobs
• To suspend the currently active program, use

• To return to the program you just suspended,
  type “fg”

• To put the program you just suspended in the
  background, type “bg”
To see a list of your programs
      running, type “ps”.

  PID TTY         TIME CMD
  866 pts/1   00:00:00 tcsh
  872 pts/1   00:00:00 ps
   use kill to end a process
 PID   TTY         TIME   CMD
 866   pts/1   00:00:00   tcsh
 874   pts/1   00:00:00   cat
 875   pts/1   00:00:00   ps

>kill 874
[1]    Terminated    cat
Linux Commands

               UNIX Commands
      A command is a program which interacts with the kernel
      to provide the environment and perform the functions
      called for by the user.

      A command can be: a built-in shell command; an
      executable shell file, known as a shell script; or a source
      compiled, object code file.

      The shell is a command line interpreter. The user
      interacts with the kernel through the shell. You can write
      ASCII (text) scripts to be acted upon by a shell.
Linux Commands

                     UNIX Shell
     The shell sits between you and the operating system,
     acting as a command interpreter.

     It reads your terminal input and translates the
     commands into actions taken by the system. The shell is
     analogous to in DOS.

     When you log into the system you are given a default

     When the shell starts up it reads its startup files and may
     set environment variables, command search paths, and
     command aliases, and executes any commands specified
     in these files.
Linux Commands

                      UNIX Shell
   The original shell was the Bourne shell, sh.
   Every Unix platform will either have the Bourne shell,
   or a Bourne compatible shell available.
   The default prompt for the Bourne shell is $ (or #, for
   the root user).
   Another popular shell is C Shell. The default prompt
   for the C shell is %.
Linux Commands

                    UNIX Shell
      Numerous other shells are available from the
      network. Almost all of them are based on either sh or
      csh with extensions to provide job control to sh,
      allow in-line editing of commands, page through
      previously executed commands, provide command
      name completion and custom prompt, etc.
      Some of the more well known of these may be on
      your favorite Unix system: the Korn shell, ksh, by
      David Korn and the Bourne Again SHell, bash, from
      the Free Software Foundations GNU project, both
      based on sh, the T-C shell, tcsh, and the extended C
      shell, cshe, both based on csh.
Linux Commands

            Shell Programming
     You can write shell programs by creating scripts
     containing a series of shell commands.

     The first line of the script should start with #! which
     indicates to the kernel that the script is directly

     You immediately follow this with the name of the shell,
     or program (spaces are allowed), to execute, using the
     full path name. So to set up a Bourne shell script the first
     line would be: #! /bin/sh
Linux Commands

              Shell Programming
     The first line is followed by commands
     Within the scripts # indicates a comment from that point
     until the end of the line, with #! being a special case if
     found as the first characters of the file.
     cd /tmp
     mkdir t
     You also need to specify that the script is executable by
     setting the proper bits on the file with chmod, e.g.:
     $ chmod +x shell_script
Linux Commands

    File Management and Viewing
    Filesystem Mangement
    Help, Job and Process Management
    Network Management
    System Management
    User Management
    Printing and Programming
    Document Preparation
Linux Commands

          Command Structure
    Command <Options> <Arguments>
    Multiple commands separated by ; can be
    executed one after the other
Linux Commands

     Help Facilities for Commands
   To understand the working of the command and
   possible options use (man command)
    Using the GNU Info System (info, info command)
   Listing a Description    of   a   Program   (whatis
   Many tools have a long−style option, `−−help', that
   outputs usage information about the tool, including
   the options and arguments the tool takes. Ex:
   whoami --help
Linux Commands

   An important early development in Unix was the
   invention of "pipes," a way to pass the output of
   one tool to the input of another.
            eg. $ who | wc −l
   By combining these two tools, giving the wc
   command the output of who, you can build a new
   command to list the number of users currently on
   the system
Linux Commands

Linux File Management and Viewing
  File and Directory management

    cd Change the current directory. With no arguments "cd" changes
    to the users home directory. (cd <directory path>)
    chmod Change the file permissions.
    Ex: chmod 751 myfile : change the file permissions to rwx for owner,
    rx for group and x for others
    Ex: chmod go=+r myfile : Add read permission for the group and
    others (character meanings u-user, g-group, o-other, + add
    Ex: chmod +s myfile - Setuid bit on the file which allows the
    program to run with user or group privileges of the file.
Linux Commands

Linux File Management and Viewing
  There are three such special permissions within Linux. They are:
     setuid — used only for applications, this permission indicates that the
     application is to run as the owner of the file and not as the user executing
     the application. It is indicated by the character s in place of the x in the
     owner category. If the owner of the file does not have execute permissions,
     the S is capitalized to reflect this fact.
     setgid — used primarily for applications, this permission indicates that the
     application is to run as the group owning the file and not as the group of
     the user executing the application. The setgid permission is indicated by
     the character s in place of the x in the group category. If the group owner of
     the file or directory does not have execute permissions, the S is capitalized
     to reflect this fact.
     sticky bit — used primarily on directories, this bit dictates that a file
     created in the directory can be removed only by the user that created the
     file. It is indicated by the character t in place of the x in the everyone
     category. If the everyone category does not have execute permissions, the
     T is capitalized to reflect this fact.
Linux Commands

Linux File Management and Viewing
   chown Change owner.
   Ex: chown <owner1> <filename> : Change ownership
   of a file to owner1.
   chgrp Change group.
   Ex: chgrp <group1> <filename> : Change group of a file
   to group1.
   cp Copy a file from one location to another.
    Ex: cp file1 file2 : Copy file1 to file2
    Ex: cp –R dir1 dir2 : Copy dir1 to dir2
   md5sum Prints the MD5 Checksum
Linux Commands

Linux File Management and Viewing

   ls List contents of a directory.
    Ex: ls, ls –l , ls –al, ls –ld, ls –R
   (-rwxrwxr-x 1 juan juan 0 Sep 26 12:25 foo )
   |more will list page wise
   mkdir Make a directory.
    Ex: mkdir <directory name> : Makes a directory
   Ex mkdir –p /www/chache/var/log will create all the
   directories starting from www.
   mv Move or rename a file or directory.
    Ex: mv <source> <destination>
Linux Commands

Linux File Management and Viewing
    find Find files (find <start directory> -name <file name> -print)

    Ex: find /home –name readme -print
    (Search for readme starting at home and output full path.)
    ―/home" = Search starting at the home directory and proceed
    through all its subdirectories
    "-name readme" = Search for a file named readme
    "-print" = Output the full path to that file

    locate File locating program that uses the slocate database.
    Ex: locate –u to create the database,
        locate <file/directory> to find file/directory
Linux Commands

Linux File Management and Viewing
    pwd Print or list the present working directory with full
    rm Delete files (Remove files). (rm –rf <directory/file>)
    rmdir Remove a directory. The directory must be empty.
    (rmdir <directory>)
    touch Change file timestamps to the current time.
    Make the file if it doesn't exist. (touch <filename>)
    whereis Locate the binary and man page files for a
    command. (whereis <program/command>)
    which Show full path of commands where given
    commands reside. (which <command>)
Linux Commands

Linux File Management and Viewing

  File viewing and editing
    emacs Full screen editor.
    pico Simple text editor.
    vi Editor with a command mode and text mode. Starts in
    command mode.
    gedit GUI Text Editor
    tail Look at the last 10 lines of a file.
    Ex: tail –f <filename> ,
    Ex: tail -100 <filename>
    head Look at the first 10 lines of a file. (head
Linux Commands

Linux File Management and Viewing
  File compression, backing up and restoring
    compress Compress data.
    uncompress Expand data.
    cpio Can store files on tapes. to/from archives.
    gzip - zip a file to a gz file.
    gunzip - unzip a gz file.
    tar Archives files and directories. Can store files and
    directories on tapes.
     Ex: tar -zcvf <destination> <files/directories> - Archive
    copy groups of files. tar –zxvf <compressed file> to
    zip – Compresses a file to a .zip file.
    unzip – Uncompresses a file with .zip extension.
Linux Commands

Linux File Management and Viewing
    cat View a file
    Ex: cat filename
    cmp Compare two files.
    cut Remove sections from each line of files.
    diff Show the differences between files.
    Ex: diff file1 file2 : Find differences between file1 & file2.
    echo Display a line of text.
Linux Commands

Linux File Management and Viewing

   grep List all files with the specified expression.
   (grep pattern <filename/directorypath>)
   Ex: ls –l |grep sidbi : List all lines with a sidbi in them.
   Ex: grep " R " : Search for R with a space on each side
   sleep Delay for a specified amount of time.
   sort Sort a file alphabetically.
   uniq Remove duplicate lines from a sorted file.
   wc Count lines, words, characters in a file. (wc –c/w/l

Shared By: