shellscripting by keralaguest


									                                    System Administration
                                      Course Notes #17

Shell Scripts
You have already examined shell scripts in various labs. A shell script is a program that, when
run, is interpreted in the Bash shell. This means that the program is not compiled, but instead
each instruction is read by the shell interpreter and executed as if you were typing the commands
in from the command prompt. Shell scripts can be as basic as a sequence of Linux commands or
as complex as Java programs with input and output, loops, nested if-else clauses, variables
(including arrays but not objects), functions (methods) and parameters. You will find some
aspects of Bash shell scripting to be easy since you already know how to program, but you will
most likely find the syntax very awkward (the Bash shell is based on Algol syntax, a
programming language from around 1960!)

Why would you write a shell script? There are many tasks that you might face as a system
administrator that you want to automate. By capturing the sequence of commands in a script,
you can run the script any time you want (or you can invoke it automatically using at or crontab).
You might, for instance, write a script that takes a textfile of people‟s names and their classes for
the semester and automatically generate class accounts for every student on a Linux machine.
Or, you might write a script that searches all of the files in /home to see if any have bad
permission settings (such as 000 or 777). Or, you might write a script that searches some of the
log files for unusual activity (such as a number of failed log in attempts which might indicate
someone trying to obtain unauthorized access).

Before we get into the specifics of shell scripting, note that you can write scripts in many
different languages such as php, Javascript, Ruby or perl. The language you are using is that of
the Bash shell, so we might call them Bash scripts. If you were using a different shell, your
script would be in a different language with different syntax. The csh (c-shell) for instance uses
syntax that looks like C.

Each script will be placed in its own file. To run the script, you must make sure that the script‟s
permission is executable. To run it, you use the syntax ./name where name is the filename
containing the script. If the script permits parameters, you can list them after the name as in
./name file.txt –c or something like that. All Bash scripts must start with the line #!/bin/bash
which tells the Bash interpreter that what follows should be executed by the Bash interpreter,
which is located in /bin. Note that # is used for comments, so any comment you want to add will
start with #.

Here, we will cover many of the things you will need to know to write your own scripts. These
set of notes will be used over two sessions. You should read the entire set of notes before
starting either of the two labs, and then have the notes accessible while you work through both
labs and the accompanying homework.

Just as in other programs, you can have variables in a shell script. There are two differences
between a script variable and a variable as found in languages like C or Java: first, you do not
declare your variable but instead just assign it when you are ready to use it, and second, when
you want to access the value stored in a variable, precede the variable name with a $. For
        I=0                   // initializes I to 0
        J=$I                  // sets J to equal the value stored in I
        E=$((C+D))            // when using a variable in an expression, you surround the
                              // expression with $((…))
        X=$((X+1))            // here, we increment the value in variable X

In addition to variables that you define in your program, you can also use variables that have
been defined in the shell that you are operating in. For instance, $USER and $PWD as defined
in your .bashrc, or a variable that you define at the command line prompt. However, to use one
of these variables, it must first be exported. An example is shown later.

Examine your .bashrc to see other variables defined. If you define a variable at your command
line prompt, it is also available for any shell scripts you run before you close that shell or log off.

Output Statements

The basic output statement is echo, much as you used early in the semester. The echo statement
can combine literal text with variables and the results of commands. For instance, you might
have the following code:
       echo Hello $Name, the date and time are `date`
Notice the use of $ for Name, and surrounding date with `` as we did earlier in the semester.

Here is an example script, we will call it script1:
        echo $X $Y $Z

Once we write script1, we need to change its permissions to be executable, so from the command
line, I might do the following:
         $ chmod 755 script1
         $ ./script1
         0 1 1                      // the script‟s output

Now consider the following modified version of script1:
       echo $Z $A

A has not been defined in our script. We will define it from the command line.
        $ A=5
        $ ./script1
        1 1
Why did A output 1 when we assigned it 5? Shouldn‟t the output be 1 6? Actually, no, A in the
shell script is not the same as A from the command line because we didn‟t export A. So we
continue by doing:
        $ export A
        $ ./script1
        1 6
And now we get the right output.

Here is another sample script, we will call it script2:
        echo Hello $USER, you are currently at $PWD

Now, we do the following from the command line:
      $ chmod 755 script2
      $ ./script2
      Hello zappaf, you are currently at /home/zappaf

Input Statements

The input statement is read. As with other programming languages, you specify the variable(s)
after the read statement so that the input is stored in the accompanying variable(s). Typically,
you will only read in a single value in your read statement, so you would use read name,
where name is the name of your variable. If you enter a numeric value, the variable will store
the number, if you enter something comprised of any other characters, the variable will store a
string (even if the input starts with numbers). As the read statement merely waits for you to
enter something via keyboard, you will often want to precede your read statement with a
prompting message using echo. Here are two examples:

       echo Enter a positive number
       read num
       echo Enter your first name
       echo fname

To use num or fname, recall that you will need to add a $ before the variable name as in $num.
Assignment Statements

A script assignment statement will look like this var=expr where var is a variable and expr is
some mathematical expression. However, recall that if a variable is referenced in the expression,
it is preceded by a $. If the expression contains more than just a variable, you must use $((…)).
You can also use $[…] if you prefer. Here are some example assignment statements:
         I=$J                  // sets I to the value stored in J
         J=0                   // sets J to 0
         I=$((I+1))            // increment I
         avg=$((sum/count)) // compute the average of count values that total to sum
         z=$(((x+y)/(x-y)))    // expression requires its own ( ) to force order of operations
         z=$[(x+y)/(x-y)]      // another way to write the previous instruction
NOTE: variables in the Bash Shell can only store integers or strings, but not reals/floats. So, for
a division operation, you only get the whole portion of the quotient. To obtain the remainder,
use % (the mod operator as found in Java or C). For instance, if x=12 and y=5, then x/y is 2 and
x%y is 3 (12/5 has a quotient of 2 with a remainder of 3).


If our shell scripts consisted of only input, output and assignment statements, it would perform
the same way every time we would run it. Most programs require that we have control
statements as well, to control what instructions get executed and how many times. We use
selection statements (if-then, if-then-else, nested if-then-else) to select which statement(s) to
execute, and iteration statements or loops (while, for) to control how many times statements get

When using a while loop or if-then statement, you will want to specify a condition. Conditions
are placed inside of [ ] marks. There are three general forms of conditions:
     Numeric comparisons, which use –eq, -ne, -gt, -lt, -ge, -le for equal, not equal, greater
        than, less than, greater than or equal to and less than or equal to respectively
     String comparisons, which use = or != for equal to and not equal to respectively
     File testing, which allows you test if a file exists, is readable, etc (as shown below)
                -d – is the item a directory?
                -e – does the file already exist?
                -f – does the file exist and is it a regular file?
                -h (or –L) – is the item a symbolic link?
                -r – does the file exist and is it readable?
                -w – does the file exist and is it writable?
                -x – does the file exist and is it executable?

Examples:      [ $x –gt 0 ] // is x greater than 0?
               [ $Name = Frank ] // does the name store “Frank”?
               [ -d $i ] // is the value of $i the name of a directory
The syntax is tricky as the interpreter requires a space before and after each item in the [ ]. So
for instance, you must use [ $x –gt 0 ], not [$x –gt 0] and you must use [ -d $i ] instead of [-d $i]
or [ -d$i ].

You can combine conditions using AND, OR and NOT. For not, use ! at the beginning of the
condition as in [ ! –e $f ] for does this file not exist? If you want to use NOT for –gt, just change
your condition to –le, and to use NOT for =, just use !=.

For AND and OR, use && and || (this is the same as in C or Java). But, if you are going to use
one of these, you MUST wrap your condition inside an additional set of [ ] as in [[ -r $f && -x
$f ]] which tests to see if file $f is both readable and executable or [[ $x –le 100 || $x –ge 1 ]] to
test to see if x is between 1 and 100.

The if-then Statement

Syntax: if [ condition ]; then action(s); fi
Or:     if [ condition ]

The syntax can be tricky. Notice that you can omit the ; after the condition if you place each
item on a separate line (other than the if and condition). The same is true of the actions. If there
are multiple actions, separate them by ; on the same line, or just put each on separate lines. The
statement must end with a fi. Forgetting the fi will most likely lead you to other errors.

Here is an example written two ways:
        if [ $FOO –ne 0 ]; then echo $FOO; fi
        if [ $FOO –ne 0 ]
                      echo $FOO

NOTE: the indentations are strictly to enhance readability and not necessary, but useful.

Two additional examples are:
         if [[ $FOO –gt 0 && $FOO –lt 10 ]]; then echo $FOO is within bounds; fi
         if [ $FOO –gt 0 ]; then FOO=$((FOO-1)); echo $FOO; fi
It is recommended that you separate the words then, the if clause, and fi so that you do not have
to worry about where to place your semicolons.

We can use if statements to help with error checking. Earlier, we used division in some
assignment statements, but if we divide by 0, we will get an error. So we could modify our code
as follows:
        if [ $count –ne 0 ]; then avg=$((sum/count)); fi
The if-then-else Statement

In our previous example, what happens if count is 0? Do we want an alternate action? If so, we
use the if-then-else statement instead of the if-then statement.

Syntax: if [condition ]; then action(s); else action(s); fi
Or:     if [ condition ]

This is almost identical to the if statement except that after the last then statement, you add the
word else and list one or more statements for your else clause.

        if [ $FOO -gt 0 ]; then FOO=$((FOO-1)); else FOO=1; fi
        if [ $FOO –gt 0 ]

We could rewrite our previous average example as
      if [ $count –ne 0 ]; then avg=$((sum/count)); else avg=0; fi
      if [ $count –ne 0 ]

Here is an example that we might actually use as system administrators. This instruction will
test to see if the file is executable and if not, will change its permission to be executable (by the
owner only)
         if [ -x /home/foxr/foobar.txt ]; then echo file is already executable; else chmod 700
         foobar.txt; fi

Alternatively, using ! we can simplify the above statement:
       if [ ! –x /home/foxr/foobar.txt ]; then chmod 700 foobar.txt; fi
The Nested if-then-else Statement

What if we have more than just two possible outcomes (true or false) such as if we want to
assign a letter grade based on the student‟s score being >= 90, between 80 and 89, between 70
and 79, between 60 and 69, or < 60? We place if-then-else statements inside of other if-then-else
statements. The resulting structure is known as a “nested” if-then-else statement.

Syntax: if [condition ]; then action(s); elif [ condition ]; then actions(s); elif [ condition ]; then
actions(s); else actions(s); fi

The alternative syntax will look like this:
       if [ condition ]
               elif [ condition ]
               elif [ condition ]

You can have as many elif clauses as you like. There is only one fi statement for the entire

Here is a simple example:

       if [ $FOO –lt 0 ]; then echo cannot take square root; elif [ $FOO –eq 0 ]; then echo 0; else
       echo too lazy to compute square root do it yourself; fi
       if [ $FOO –lt 0 ]
                        echo cannot take the square root
               elif [ $FOO –eq 0 ]
                        echo 0
                        echo too lazy to compute square root do it yourself

Here is how we can assign a letter grade based on a student‟s score.

       echo Enter student score:
       read score
       if [ $score –ge 90 ]
                then echo Student Grade is A
       elif [ $score –ge 80 ]
                then echo Student Grade is B
       elif [ $score –ge 70 ]
                then echo Student Grade is C
       elif [ $score –ge 60 ]
                then echo Student Grade is D
       else echo Student Grade is F

It is common for a student to solve this as follows instead:

       echo Enter student score:
       read score
       if [ $score –ge 90 ]
                then echo Student Grade is A
       elif [[ $score –ge 80 && -lt 90 ]]
                then echo Student Grade is B
       elif [[ $score –ge 70 && -lt 80 ]]
                then echo Student Grade is C
       elif [ $score –ge 60 ]
                then echo Student Grade is D
       else echo Student Grade is F

However, we do not need to use the && and compound conditions above because, to reach the
first elif statement, score must not be >= 90, so we don‟t need to test –lt 90.

The for Loop

Syntax: for var in list; do statement(s); done
Or:     for var in list

Again, indentation is not needed but helpful. The var is any variable name you wish to use. You
may reference the variable in your statement(s) by using $var. For instance, for i in 1 2 3 4; do
echo $i; done will output each number from 1 to 4 on different lines. As with the if statements,
after the word do, you can list multiple statements. If placed on a single line, the statements
must end with ; so that the bash interpreter knows when the next statement begins. If you
separate them onto other lines, they do not need the ;.

While we can use the for loop as shown above to work through a list of values, we will more
commonly use it to work through either a list of files or through a list of values input. We will
explore the latter later in this news.
To iterate through a list of files, we can specify a directory in the in statement, as in for x in
/home/foxr do. Or, if we just specify *, it works through the current directory. We can also use
*.txt, or some other regular expression. The following example will print all of the .txt files in
the current directory. It is, in essence, doing ls *.txt.
         for x in *.txt
                 echo $x
Lets enhance the above code also output the number of .txt files that are writable. How do we do
this? We first have to add a counter variable to the program. We will initialize it to 0. We will
call it count. We initialize it before the for loop (if we initialize it to 0 in the for loop, then count
will become 0 every time we go through the loop). Next, we need to add an if statement to test if
the current file is writable. If it is, we add 1 to count. Finally, we need to output count, but we
will do this after the for loop or else it will output the message every time through the loop. Here
is our enhanced script:
         for x in *.txt
                 if [ -d $x ]
                 echo $x
         echo The number of .txt files found in this directory that are writable is $count

A common set of code found in various shell scripts is to first test to see if a file exists, before
doing something with or to that file. Below is code that will work through the current directory
to see which files are not regular.

        for x in *
                if [ ! –f $x ]
                          echo “$x is not a regular file”

This set of code changes any non-executable files in your directory to have 700 permission (rwx
for owner, no access for anyone else)
       for x in *.txt
               if [ ! –x $x ]
                        then chmod 700 $x
While loop

Syntax: while [ condition ]; do statement(s); done

This loop will test a condition to determine whether to execute the statements (the loop body) or
not. The loop continues to execute the loop body while the condition is true. Once the condition
becomes false, the loop ends and the next instruction in the script is executed. If the condition is
never true to begin with, then the loop body is never executed. Note that the for loop allows you
to iterate through a list such as the contents of the current directory, but the while loop doesn‟t
permit that feature. Here is an example of using the while loop that uses the read statement.

echo Enter a filename, quit to exit
read file
while [ $file != quit ]
        ls –l $file
        echo Enter a filename, quit to exit
        read file

Notice here that we repeat the echo and read statements. Why? Without the read inside of the
loop, we would have an infinite loop. The echo inside of the loop instructs the user what to enter
otherwise, after outputting the ls –l of the file, the user would see the cursor blinking without
knowing what to do next or why the script is waiting.

We will most likely use a while loop to coincide with input (from the user or a disk file) whereas
the for loop will be used to iterate through a list of values instead.

Case statement

Usually, if we have multiple conditions/actions, we will use a nested if-then-else structure. But
in some cases, when we want to test a variable against a list of specific values, we can use the
case statement. For instance, if the user inputs a choice between 1 and 4 and I want to do a
different action if the input is 1, 2, 3 or 4, I can use the case instead of the nested if-then-else.
Here is what the case statement looks like:

Syntax: case $var in value1) statement(s); value2) statement(s); value3) statements; esac

And a simple example:
       case $FOO in
              1) FOO=$((FOO+1)); echo FOO is now $FOO;;
              2) FOO=0; echo FOO is now $FOO;;
              3) echo FOO is 3;;
Notice the peculiar way that we list the test values (value followed by close paren) and the way
we end each group of statements with an additional ;. You can use if-elif-else statements instead
of case.

Using Parameters

A shell script can receive parameters, just like a normal Linux command. For instance, if you do
rm /home/zappaf/myfile1.txt, the /home/zappaf/myfile1.txt is a parameter passed to the rm
program. You can write your script to receive any number of parameters. You can also test to
see what values they are, for instance if you want to write a script to receive parameters like –a
or file names or other types of things.

Inside the shell script, a parameter is accessed by using $n where n is the numeric placement of
the parameter. The first parameter is $1, the fifth parameter would be $5 and so forth. $# gives
you the number of parameters and $@ returns all of the parameters in a list. The following script
would output each parameter on a separate line.

for i in $@
         echo $i

If this was in a script called foo and you invoked it as ./foo a b c d, the output would be a, b, c
and d on separate lines. The following code would perform an ls –l on each filename provided.

for name in $@
       ls –l $name

Imagine that a shell script required at least one parameter. You can test to see if any parameters
were passed to the script by using
       if [ $# -gt 0 ]
The condition is true if $# (the number of parameters) > 0. So you might have code like this:
       if [ $# -gt 0 ]
                        … // do whatever operations are needed
                        echo Illegal use of this script, at least one parameter is expected!

Let‟s write a script that will add two numbers together and output the sum.
       if [ $# -eq 2 ]
                        echo $((1+2))
               echo Error in input, expecting two parameters and received $#

The following code will output whether each of a list of file names is of an existing, regular file.

       if [ $# -eq 0 ]
                         echo Error, no parameters passed
                         echo The following are names of files that exist and are regular
                         for i in $@
                                       if [ -f $i ]
                                                then echo $f

Accessing Files

There are a couple of ways to access information in files from a script. First, you can pass the
filename as a parameter and then use the cat instruction to obtain each string in the script and
process them all in a for loop. We can also access all items in a file using the while read
statement (see the next section).

As an example, assume a file, numbers.txt, consists of a list of integer numbers. We write the
following script, called counter, and run it with ./counter numbers.txt

Notice that the script makes sure that the file is valid, and also makes sure that the file has some
numbers in it.

       if [[ ! –f $1 || ! –r $1 ]]
                          echo $1 is not a valid or readable file
                          for value in `cat $1`
                          echo Number of values is $count
                          echo Sum of values is $sum
                          if [ $count –gt 0 ]
                                      echo Average of values is $((sum/count))
                                      echo File empty, cannot compute average

The while read Statement

If the file being input consists of different types of values, we want to use a different approach.
Assume we have a file that consists of records (rows) where each record is a student‟s name,
major, GPA and number of hours earned to date. We want to compute the average hours for all
CSC and CIT majors. The while read statement can be used. The syntax for while read is while
read field1 field2 field3 where field1, field2, field3 are the names of the values we will read in.
In our example here, these will be name, major, gpa and hours. The following code will
compute the average for us.

       while read name major gpa hours
                       if [[ $major = CIT || $major = CSC ]]
       if [ totalStudents –gt 0 ]
                       echo Average hours of CIT/CSC majors is $((totalHours/totalStudents))
                       echo There were no CIT or CSC majors

If this script was called avg_hours and the file was class_list.txt, we would invoke this as
./avg_hours < class_list.txt.

What if we wanted to do the same for the average GPA for our students? It would seem sensible
just to change hours to GPA and totalHours to totalGPA, but remember that the Bash interpreter
cannot handle real (float) values, so you would wind up with run-time errors.

As another example, imagine that we have a list of usernames in a file and we want to find out
which ones have directories in /home (as opposed to say /home/CIT370/ or some other location).
How do we do this? We use the while read to get each name from the file and then we test to see
if /home/$name exists using –d. The code follows.
       while read name
                     if [ ! –d /home/$name ]
                                     echo $name

If the shell was called directory_checker, and our file of students is students.txt, we would
invoke the shell as ./directory_checker < students.txt.

What if we wanted to see if the student listed in the file has an account and does not have a
/home directory, and if so, create that directory? Well, this is a little trickier. We can test to see
if $name has a directory in /home as shown above. But how do we know if $name actually has
an account? Recall that all users‟ username are listed in /etc/passwd. We can use grep on $name
to see if it exists in /etc/passwd. But then what? The grep command will either return the line
where it found the entry, or an empty line. What if we piped the result of grep to wc –l to count
the number of responses? It should either be a 0 (if $name doesn‟t exist in the file) or 1 (if
$name does exist). So we add the following prior to the if statement in our script
        exists=`grep $name /etc/passwd | wc -l`
We then alter our if statement to test to see if exists is 1 and ! –d /home/$name is true. If both
are true, then $name is a real account but without a directory in home. The if statement will look
like this
        if [[ $exists –ne 0 && ! –d /home/$name ]]
Finally, instead of doing echo $name as our action, what we want to do is create a directory for
$name in /home, so our statement changes from echo $name to mkdir /home/$name. We may
also want to state the operation that took place so that the system administrator knows. But
imagine that we run this shell script every week using crontab. Instead of outputting the result to
the screen, lets output the resulting action to a log file called home_dir_new_users.txt. Our final
script will look like this:
        while read name
                         exists=`grep $name /etc/passwd | wc –l`
                         if [[ $exists –ne 0 && ! –d /home/$name ]]
                                         mkdir /home/$name
                                         echo “$name directory created in /home”
                                                >> home_dir_new_users.txt
Using Advanced Linux Operations

Here, we look at combining shell scripting with the awk command. Consider for instance that
you want to compute the sum of the size of all files that are readable in a directory. How can we
do this? We can output the names of all readable files by using for i in * combined with if [[ -f
$i && -r $i ]] then echo $i. But how do we get the file sizes? If we use echo `ls –l $i` in place of
echo $i, then we get a long listing of the file information, but still this does not sum up the values
we want. Recall that awk can return (print) a particular field from the input it matched. We can
use awk to do a {print $5} since the file sizes are the 5th column of an ls –l. The awk statement
will look like this: awk „{print $5}‟ but rather than specifying a filename, we want to pipe the ls
–l $i to the awk statement. So now, our statement will look like this: `ls –l $i | awk „{print
$5}‟`. Notice how this ends, with ‟ to end the awk statement and ` to end the “this is a Linux
command to execute”. Now what do we do with the value returned from awk (which is the
size)? We add it to a running total. Here is the code that will perform this entire operation:

       for i in *
                if [[ -f $i && -r $i ]]
                                temp=`ls –l $i | awk „{print $5}‟`
       echo The readable files in this directory are $sum in size

Note that we did not need temp but instead could have added the statement in ` ` into the
assignment statement as sum=$((sum+`ls –l $i | awk „{print $5}‟`)) but that syntax is very ugly
and awkward. We can also use grep (or egrep) to test for patterns. For instance, if we want to
find all files whose pattern is –rwxrwxrwx and place the file names in a log file so that we can
later tell owners to correct the permission, we might use code like this:

       for file in *
                       if [ -f $file ]
                                         perm=`ls –l $file | awk „{print $1}‟`
                                         if [ $perm = „-rwxrwxrwx‟ ]
                                                         echo $file >> bad_permission_files.txt
Here, we test each file to see if it is an existing file and if so, we obtain its permissions using
awk. Now we compare this string against –rwxrwxrwx. If the permission matches, we echo the
file name and append it to the file bad_permission_files.txt.

One thing to recall about awk is that it actually has a built-in loop. That is, it will iterate through
all of the input for us. So we could actually accomplish the same task with

       ls –l | awk „/-rwxrwxrwx/ {print $9} >> bad_permission_files.txt‟

The only flaw with the above code is that it doesn‟t test to see if each file tested is truly a file
(i.e., using –f as we did in the previous script). We have no way of including the condition –f
here since awk is iterating through all files and not testing a specific file. So we can use awk in
some cases, but not always.

Note that in both of these scripts, we are appending the output to a specific file that we hard-
coded into the script. We do not necessarily have to limit a script to this action. Instead, we
could allow the user to specify the filename by using >> at the command-line prompt when
calling the script. If the filename is included as a parameter, we would use echo $file >> $1. We
could write our echo statement like this:
        if [ $# -gt 0 ]
                         echo $file >> $1
                         echo $file >> bad_permission_files.txt

Alternatively, we could just leave the echo statement as echo $file with no redirection of the
output and then the user can either see the output on the screen, or redirect the output to a file by
using something like ./script >> bad_permission_files.txt.

Consider a script that calls for an input file to be used with a while read statement, and then the
user wants to redirect the output. The syntax to invoke such a script would look like this:
       ./script < inputfile > outputfile

To top