Introduction to Bash Programming

Document Sample
Introduction to Bash Programming Powered By Docstoc
					Review: filters, bash
   programming
 CSRU3130, Spring 2008
     Ellen Zhang



                         1
             Announcements
• Homework 4 due today
• Makeup class: exercise on shell programming
  – Tuesday, Feb 26, 1pm-2:15pm, Computer lab
  – Wednesday, Feb 27, 10am-11:15am, Computer lab
• Midterm exam:
  – Friday, Feb 29
  – Close book, ok to bring a one-page cheatsheet


                                                    2
       Course works: expectation
• At least 3 hours studies each week
   – After each class: review slides, make sure you understand
     and remember important concepts
      • Command substitution, filename expansion, command line
        arguments, standard input/output/error …
      • Shell loop structure, branch structure, …
   – At least once a week: read textbook and online tutorial
   – Homework/Exercise: experiment with computers, learn by
     trying things out




                                                                 3
       For the rest of the course
• Regular quizzes
• More in-class exercises, questions
• Final grades
  – Will reflect not only how well you did, but also
    how hard you tried too…




                                                       4
                      Outline
• Bash programming review
  – Homework 3 Q1, Q2, Q4
• grep family and regular expression
  – Homework 3 Q5
• More filters:
  – sed
  – sort, tr, cut, paste, comp, uniq


                                       5
                      Last class
• Shell’s filename expansions (pattern matching)
  – rm *, ls *
  – ls {a,b}.o, ls ?.o, ls [a-z].o
• Command substitution: substitute with
  standard output of a command
  – file_num=`ls -l | wc –l`
  – cur_time=$(date)$
  – Or used in a script:
  for student in `cat students.txt`
  …                                            6
               set command (1)
• When invoked without arguments, set shows
  values of environment variables
  – set | grep PATH
  – echo $PATH
• If followed by ordinary arguments (no options
  that starts with -), reset values of $1,$2, …
  – set Hello world    # $1 is set to “Hello”, $2 is “world”
  – set “Hello World!”
  – set `ls –l | wc `


                                                               7
             set command (2)
• When invoked with options, set turns on/off
  shell options
  – -f : disable filename expansion (globbing)
  – -v: Print shell input lines as they are read.
  – -x: Print a trace of simple commands and their
    arguments after they are expanded and before
    they are executed.
• To unset these options, use + instead of –
  – set +f

                                                     8
         Tracing shell commands
• Display commands after they are expanded:
[zhang@storm Demo]$ set –x
[zhang@storm Demo]$ ls *
+ ls --color=tty b.o CCodes chdir dd Examples exercise1 fe
   maillist.txt pick …
b.o chdir dd fe maillist.txt pick README README1 subshell
   subshell_nopar test
++ echo -ne '\033+0;zhang@storm:~/Demo‘




                                                             9
                            Examples
• Let’s see how pick script is executed
  #!/bin/bash
  set -x
  for i                     #Same as : for i in $*
  do
        echo -n "$i (y/n)" 1>&2
        read input
        if [ $input = 'y' ]
        then
              str="$str $i"
        fi
  done
  echo $str
                                                     10
                  Tracing pick script
[zhang@storm Demo]$ echo "red        + '[' n = y ']'
    green blue yellow" > color.txt   + for i in '"$@"'
[zhang@storm Demo]$ ./pick `cat      + echo -n 'blue (y/n)'
    color.txt`                       blue (y/n)+ read input
                                     n
+ for i in '"$@"'
                                     + '[' n = y ']'
+ echo -n 'red (y/n)'                + for i in '"$@"'
red (y/n)+ read input                + echo -n 'yellow (y/n)'
                                     yellow (y/n)+ read input
y
                                     y
+ '[' y = y ']'                      + '[' y = y ']'
+ str=' red'                         + str=' red yellow'
                                     + echo red yellow
+ for i in '"$@"'
                                     red yellow
+ echo -n 'green (y/n)'              [zhang@storm Demo]$
green (y/n)+ read input
n
                                                                11
        Homework 3 Question 1
• a script that prints out all empty files (fe).
• Syntax: fe path_name
• Functions:
   – check all files located under given directory, and
     print to the standard output the list of empty files.
      • e.g. suppose in the directory, there are two empty files,
        a.o, and b.o, then ./fe will print:
          a.o b.o.



                                                                12
      Hw 3 Question 1: find out
        commands needed
• What we need to do ?
  – Find out files under given directory
  – For each file, check whether it is a regular and is
    empty ; if so, prints its name to standard output
     • Go to the course web page , Advanced bash-scripting
       guide, Part 2, 7 Tests, 7.2 File test operators
        test –f fileName
        test –s fileName
     • Put them together ?


                                                             13
              AND list construct
• Statement1 && statement2 && statement3 && …
  – first statement is executed
  – If it returns false, return false; if it returns true, next
    statement is executed…. Until finding first false statement,
    or all statements return true
• For example:
  – test –f myFile && test –s file
  – If myfile is not regular file (i.e., first statement returns
    false), then the second test command is not executed and
    the who list return false

                                                                   14
                  AND list construct
• Statement1 && statement2 && statement3 …
• Statement can be:
   – test command, test –f fileName, [ -n “$string” +
   – Other commands
       • e.g. echo always returns true
       • grep returns true if found a match …
   – In a script, use exit n to return an exit code
       • 0: success; otherwise failure
   – If a script doesn’t have exit n; its return code is the return code of the
     last command in script
• Ex: * “$str” = Morning + && echo “Good $str”

                                                                              15
                 OR list construct
• Statement1 || statement2 || statement3 ..
   – first statement is executed
   – If it returns true, return true (statement2, 3 is note
     executed)
   – if it returns false, next statement is executed…. Until
     finding first statement that returns true (the construct
     then return true), or all statements return false (the
     construct then returns false).
• Ex: * “$str” = Morning + || echo “Good Afternoon”


                                                                16
       Hw 3 Question 1: put them
              together
• Use loop and branch constructs
for i in each files under given directory
                     `ls $1`
do
  if file i is empty, regular file
                                   test –f $i && ! test –s $i
  then                             [ –f $i ] && [ ! –s $i ]
     Print the i filename to standard output
  fi             echo $i

done
                                                                17
        Hw 3 Question 1: the script
#!/bin/bash
# fe: find and print out all files of zero size
#
for i in `ls $1`
do
      if [ -f $i ] && [ ! -s $i ]
      then
             echo $i
      fi
done

                                                  18
               A better solution
du $* -a | grep '^0' | cut -f2 | tr '\n' ' '

• Courtesy Michael Karp
          Hw3 Q2: find 10 largest files
• Original list10 script:
if [[ "$1" == -[0-9]* ]]
then
      for path in $*
      do
           if [ $path != $1 ]
           then
                 path_list="$path_list $path"
           fi
      done
      ls -Rl $path_list|sort -k 5 -nr | head $1
else
      ls -Rl $* | grep ^- | sort -k 5 -nr | head -10
fi
                                                       20
     Hw3 Q2: extended version
• Allow user to specify number (i.e., -n) anywhere
  – Usage: list10 ~/Documents ~/Projects -30
• How to do this ? In two steps
  – Parsing command line arguments to find out:
     • How many files user want to display ?
     • What are the directories to check ?
  – Call “ls, grep, sort, head ” with appropriate
    arguments to do the work


                                                    21
   Hw3 Q2: parsing command line
            arguments
• Goal: find out
   1. How many files user want to display ?
   2. What are the directories to check ?
   – Use variables (lineNum, pathList) to save answer to
      above questions
• Check each command line arguments
   – If it is a number argument, save it to lineNum
   – Otherwise, the argument is a path name, we want to
     append it to PathList


                                                           22
Now try to put them together




                               23
       Hw3 Q4: delete empty files
• Goal: a script, de, that
   – finds all empty files under a directory     fe script

   – for each of them, asks user yes or no       pick script
   – deletes files that user responds with yes
• Putting them together: how ?
   – rm `pick `fe pathname``
   – rm `pick \`fe pathname\``
   – rm `pick $(fe pathname)`




                                                               24
Hw3 Q4: delete empty files (cont’d)
• Putting them together
  – rm `pick \`fe pathname\``
  – rm `pick $(fe pathname)`
• What if pick returns empty string ?
  fileChosen=`./pick $(./fe)`
  if [ -n "$fileChosen" ]       test –n “$fileChosen”:
  then                          Return true if the string is not empty.
                                Always quote the string being tested !
        rm $fileChosen          See “Advanced Bash-Scripting
  fi                            Guide” for a complete list of test.




                                                                      25
                Here document
• Pass input that is embedded in script to a
  command

  #!/bin/bash
  mail $* << DELIMIT_OF_EMAIL
  This is a test email sending using
  here document using ~/Demo/test script.
  DELIMIT_OF_EMAIL

  Here document starts with <<, followed by a
  special string which is repeated at the end of the
  document.
                                                       26
                 Bundle script
• bash script, gen_bundle.sh
  – Pack files specified in command line arguments
    into a “bundle file”
     • ./gen_bundle.sh bundle.sh 411 > mybundlefile
  – Mybundlefile, a “bundle file”, is a shell script itself
     • contains commands for unpacking, and files themselves
     • When executed, it creates those files being packed




                                                          27
             Inside mybundlefile file
#To unbundle, bash this file          end of bundle.sh
echo bundle.sh                        echo 411
cat >bundle.sh <<'End of bundle.sh'   cat >411 <<'End of 411'
#!/bin/bash                           grep "$*" <<End
echo '#To unbundle, bash this file'   dial-a-joke 212-976-3838
for i in $@ ## or for i               dial-a-prayer 000000000
do                                    dial santa 8900999
       echo "echo $i 1>&2"            today is `date`
       echo "cat >$i <<'End of $i'"   end
       cat $i                         end of 411
       echo "End of $i"
done

    What happens if we run ./mybundlefile ?                      28
                      gen_bundle.sh
• For each file specified in command line
  – Print out codes for unpacking the file (to standard
    output)
  cat > FILENAME <<START of FILENAME
   Contents of FILE




  END of FILENAME

                                                      29
              gen_bundle script
#!/bin/bash
echo ‘# To unbundle, bash this file’
for i
do
      echo “echo $”
      echo “cat >$i <<‘End of $i’”
      cat $i
      echo “End of $i”
done
                                       30
          HW3 Q6: gen_bundle
            with manifest
• modify bundle so that it packs a "manifest“
  into the bundle
  – a list of archived files and other info. about them,
    i.e., those provided by “ls –l”.
  – When bundle file is unpacked, a file named
    "manifest" is created with info such as:
  # Manifest of files bundled:
  -rwxr-xr-x 1 zhang staff 104 2008-01-30 14:02 411
  -rwxr-xr-x 1 zhang staff 203 2008-02-15 11:26 bundle.sh

                                                            31
             Bundle: how to
           generate manifest ?
• We know how to obtain info. needed to for
  manifest file
  – call “ls –l” on each files being packed
• gen_bundle needs to generate bash script
  – When executed, generate a file named "manifest"
    with above info
• Use here document again !


                                                  32
           How to generate manifest
#!/bin/bash                            #generate manifest
echo ‘# To unbundle, bash this file’   echo “generate manifest”
for i                                  echo “cat >manifest <<„End of manifest‟”
do                                     for i
      echo “echo $i ”                  do
      echo “cat >$i <<‘End of $i’”         ls -l $i
      cat $i                           done
      echo “End of $i”                 echo “End of manifest”
done




                                                                        33
                     Outline
• Bash programming review
  – Homework 3 Q1, Q2, Q4, Q6
• grep family and regular expression
  – Homework 3 Q5
• Other filters:
  – sed
  – sort, tr, cut, paste, comp, uniq


                                       34
  Example of regular expressions

• Variable names in C
  – [a-zA-Z_][a-zA-Z_0-9]*
• Dollar amount with optional cents
  – \$[0-9]+(\.[0-9][0-9])?
• Time of day
  – (1[012]|[1-9]):[0-5][0-9] (am|pm)
• What does [1900-2000] match ?
  – 1998, 2000, 1, 2, 3, 4, 8, 9 … ?
  – Range only applies to single character
                                             35
      HW3 Q5: nstudents script
• How many student accounts are there on
  storm ?
  – group ID for student is 501
  – User accounts are stored in file /etc/passwd
     • walton:x:2747:501:Myasia D Walton, FAll Jan 17, 2006
       Pro. Tran:/home/students/walton:/bin/bash
     • What does each field mean ?




                                                              36
HW3 Q5: nstudents script (cont’d)
• Source of information
  – Each line in /etc/passwd corresponds to a user
  – A student account has a group id of 501
• How to find out # of student accounts ?
  – Select lines where group id field is 501
     • grep ….
  – Count number of lines using “wc –l”
  – User pipeline to connect the two:
     • grep … /etc/passwd | wc -l
                                                     37
 HW3 Q5: nstudents script (cont’d)
• Select accounts (lines) where group id is 501
   – grep 501 /etc/passwd
      • Problem: accounts with user id 2501, or group id 5501 will be
        selected too
   – grep “:501:” /etc/passwd
      • Problem: Accounts with user id 501 will be selected too
   – grep ":501:[a-zA-Z]*" /etc/passwd
      • A 501 field, followed by a field of letters (info. Field)
   – grep “^*^:+*:*^:+*:*0-9]*:501" /etc/passwd
      • Select lines where fourth field (group id) is 501, safest solution



                                                                             38
           Grep: back-references
• Find accounts whose uid==groupid
   – grep '^[^:]*:[^:]*:\([0-9]*\):\1' /etc/passwd
• Back-references
   – Use \( and \) to specify sub-expressions (i.e. tagged
     expression)
   – \n: backreference specifier, matches exactly the string that
     has matched the nth tagged expression
• e.g. search all lines where first word is same as last:
   – grep ‘^\([[:alpha:]]\{1,\}\) .* \1$’
     filename

                                                               39
         Specify pattern in files
• -f option: useful for complicated patterns, also
  don't need to worry about shell
  interpretation.
• Example
  – $ cat alphvowels
    ^[^aeiou]*a[^aeiou]*e[^aeiou]*i[^aeiou]*o[^aeio
    u]*u[^aeiou]*$
  – $ egrep -f alphvowels /usr/share/dict/words
    abstemious ... tragedious
                                                  40
                     Outline
• Bash programming review
  – Homework 3 Q1, Q2, Q4, Q6
• grep family and regular expression
  – Homework 3 Q5
• More filters
  – sed
  – sort, tr, cut, paste, comp, uniq


                                       41
      Sed: Stream-oriented, Non-
        Interactive, Text Editor
• Look for patterns one line at a time, like grep, edit
  (change) lines of the file
• Non-interactive text editor
   – Editing commands come in as command line arguments ,
     or as script
   – an interactive editor ed which accepts same commands
• For example:
   – sed ‘s/UNIX/UNIXTM/g’ filename > output
   – Change all occurrences of UNIX to UNIXTM in given files,
     and write output to file “output”

                                                                42
             Conceptual overview
 All editing commands are applied in order to each
  line in the input file
   – If a command changes input, subsequent command
     address will be applied to current (modified) line, not
     original input lines.
• Original input file is unchanged (sed is a filter)
   – By default, results are sent to standard output (but can
     be redirected to a file).
   – -n: turn off automatic printing


                                                                43
        Sed Flow of Control


                                    script


          cmd 1   cmd 2    ...    cmd n



                      print cmd

                                                      output
                      output
input                                        only without -n



                                                               44
         Address in sed command
• Specify range of lines to apply a command
  – Specified through line #:
     •   4/s/old/new/: for 4th line
     •   1,5/s/old/new/: for 1st to 5th lines
     •   5,$/s/old/new/: for from 5th line to end of file
     •   5,$!/s/old/new/: for all lines except from 5th line to end
         of file
  – Specified through pattern matching
     • /^The/s/old/new/: for lines starting with The
     • /start/,/end/s/old/new/: between first “start” to first
       “end”                                                      45
         sed command: substitute
• Substitute (s)
   – [address1[,address2]]s/pattern/replacement/[n/g]
   – Optional address fields
   – Pattern: specified using regular expression
• Example: to replace all numbers in file to XXXX:
   sed ‘s/*0-9][0-9+*/XXXX/g’ file.txt
   sed ‘/SENSITIVE_START/,/SENSITIVE_END/s/pattern/replace/g’ file.txt




                                                                         46
     sed command: substitute (2)
• Substitute (s)
   – [address1[,address2]]s/pattern/replacement/[n/g]
• n/g flag: which occurrence to apply command to
  (Pattern might occur multiple times in line)
   – Number (1,2,..) : only substitute that occurrence
   – g: substitute all occurrences within line




                                                         47
                            sed Example
• to display all online users and their login IP/host name
who
zhang pts/0        2008-02-21 17:09 (ool-44c6b5bf.dyn.optonline.net)
root pts/3        2008-02-21 10:19 (pad.cis.fordham.edu)
kerins pts/4      2008-02-21 14:15 (74.72.35.180)
portela pts/8      2008-02-21 15:33 (ool-182c2a62.dyn.optonline.net)
• Use sed to select 1st and last field
    who | sed -e 's/ .* / /' -e 's/(/ /' -e 's/)/ /‘
    zhang ool-44c6b5bf.dyn.optonline.net
    root pad.cis.fordham.edu
    ...


                                                                       48
                                  Example
• Display /etc/passwd in user-friendly manner:
    – Before:
zhang:x:3504:500:Ellen Zhang , Zhang instructional:/home/staff/zhang:/bin/bash

    – After
User: Zhang UserID: 3504 GroupID: 500 UserInfo: Ellen Zhang, Zhang instructional
   HomeDir: /home/staff/zhang Loginshell: /bin/bash
•   First step: insert “User:” to the beginning of lines
•   Second step: replace “:x:” with “ UserID:”
•   3rd step: replace the “:” before groupID to “GroupID:”
•   ...
                                                                                 49
         User friendly /etc/passwd
• First step: insert “User:” to the beginning of lines
   – sed –e ‘s/^/User: /’ /etc/passwd
  User:zhang:x:3504:500:Ellen Zhang , Zhang
  instructional:/home/staff/zhang:/bin/bash
• Replace “:x:” with “ UserID:”
   – sed -e 's/^/User:/' -e 's/:x:/ UserId:/' /etc/passwd
   User:zhang UserId:3504:500:Ellen Zhang , Zhang
     instructional:/home/staff/zhang:/bin/bash
• Replace “:” before group id with “ GroupID:”
   – sed -e 's/^/User:/' -e 's/:x:/ UserId:/' -e 's/:/ GroupId:/3'
     /etc/passwd
   User:zhang UserId:3504 GroupId:500:Ellen Zhang , Zhang
     instructional:/home/staff/zhang:/bin/bash                       50
   sed commands: delete, editing
• Delete (d): delete specified lines
  – sed ‘1,5d’ file.txt
  – sed ‘1,5!d’ file.txt
  – /<pattern>/d: delete lines containing pattern
  – Note: file.txt not touched, just the specified lines
    not displayed to standard output
• More editing commands: append, insert,
  change entire lines
  – sed ‘/secret/c\DELETED’ file.txt
                                                           51
             Sed command: print
• Print (p): print specified (selected) lines
   – sed –n ‘1,5p’ file.txt
      • Display first five lines only, same as “head -5 file.txt”
   – Sed can be used as grep:
      • sed –n ‘/<pattern>/p’ is same as grep <pattern>
• !cmd: execute the command only if line is not
  selected
   – sed –n ‘/<pattern>/!p’ is same as grep –v
     <pattern>

                                                                    52
            sed command: quit
• Quit (q): end sed session
  – sed 2/q file.txt
     • print two lines and then quit
  – sed /<pattern>/q file.txt
     • print input up to given pattern and then quit




                                                       53
                     Outline
• Bash programming review
  – Homework 3 Q1, Q2, Q4, Q6
• grep family and regular expression
  – Homework 3 Q5
• More filters
  – sed
  – sort, tr, cut, paste, comp, uniq


                                       54
                   sort command
• Basic functionalities:
   – sorts the input line by line in ASCII order
• Options
  • -k: Use the kth field (default field delimiter is space) as
    sorting key
  • -n: numeric sorting (note that "20" < "3", but 3 < 20)
  • -r: reverse order (from largest to smallest)
• Examples
   – ls –l | sort -k 5 -nr | head -10
   – ls –l | sort –k 6,7


                                                                  55
                      sort command
• Default field separator is space, use -t option to
  specify other field separator
   – Ex: sort all user accounts by uid:
       • sort -t : -k 3 -n /etc/passwd
   – Ex. sort all user accounts with groupid, and then usrid
       • sort -t : -k 4 -k 3 -n /etc/passwd
• other options
   – -f: case folding, i.e. ignore case
   – -o: output file, useful for in situ sorting
       • sort foo > foo is disastrous.
       • sort –o foo foo

                                                               56
                  uniq command
• Discard all but one of successive identical lines from
  standard input, writing to standard output
    command                          command
    command                          test
                      uniq
    test                             asd
    asd                              test
    test


• Options to change default behavior
   – -c: preceding lines with # of repetitions
   – -u: discard repeated lines (even not successive)
   – -f k: skip k fields in the comparison

                                                           57
                         Command tr
• tr: copies standard input to standard output with
  substitution or deletion of selected character
• Syntax:
   – tr [options] [set1 [set2]]
   – Input characters in set1 is translated to corresponding
     characters in set2
• Ex: Change all lower case to upper case
   – tr a-z A-Z < file



                                                               58
                       Command tr
• Options:
   – -c: complement set1
       • Apply to characters that are not in set1
   – -d: delete characters in set1, no translation
   – -s: squeeze, i.e., replace sequence of same characters
     (specified in commands) with one
• Separate words line by line:
   – tr -sc A-Za-z '\012‘             Special character: return/newline
   – tr –sc A-Za-z ‘\n’
• Ex: to replace numbers in a file with “x”
   – tr [[:digit:]] x < file | tr -s x
                                  Replace sequence of “x” with a single x
                                                                            59
                 Putting it together
Ex: Get a letter frequency count on a set of files given on
    command line. (No file names means that std input is used.)
#!/bin/bash
cat $* |
tr -sc A-Za-z '\012' |
sort |
                        Sort the words, so that same words are grouped
uniq -c |               together
sort -k 1 -nr |        Print unique words preceding with counts
head -10



                                                                         60
                          Command cut
• Cut out selected bytes or fields of each line of a file
   –    cut [-b] [-c] [-f] list [-n] [-d delim] [-s] [file]
   –    -b: select given bytes, -c: select given characters, -f: select given field
   –    -d: to specify delimiter for field
   –    list:
       1,4,7
       1-3,8
       -5,10 (short for 1-5,10)
       3- (short for third through last field))
• Ex:
   – Select user name, uid and gid from /etc/passwd
         • cut -d : -f 1,3,4 /etc/passwd
   – To show first four characters of a file
         • cut –b 1-4 file.txt
   – Show users online (list no other info)
         • who | cut –f1 –d ‘ ‘
                                                                                      61
                    Command paste
• Merge corresponding lines of files
• paste [-s] [-d list] files
     Alice          1                    Alice 1
     Betty          2                    Betty 2
     Candy          3                    Candy 3

                   paste name.txt number.txt

   – files: path name of input files
       • If - is specified, standard input will be used; the standard input will be
         read one line at a time, circularly, for each instance of -. Implementations
         support pasting of at least 12 file operands.
   – Ex. ls | paste - - - -
   – Ex. ls –l | paste - comments.txt


                                                                                    62

				
DOCUMENT INFO