Docstoc

Unix Shell Scripts - DOC

Document Sample
Unix Shell Scripts - DOC Powered By Docstoc
					    UNIX Practice Scripts:
    Basic Commands:
    Grep:
    grep -c -- Print only a count of the lines that contain the pattern.
    grep -h -- Prevents the name of the file containing the matching line from being appended to that
                line. Used when searching multiple files.
    grep -i -- Ignore upper/lower case distinction during comparisons.
    grep -l -- Print only the names of files with matching lines, separated by NEWLINE characters. Does
                not repeat the names of files when the pattern is found more than once.
    grep -n -- Precede each line by its line number in the file (first line is 1).
    grep -s -- Suppress error messages about nonexistent or unreadable files.
    grep -v -- Print all lines except those that contain the pattern.
    grep -w -- Search for the expression as a word as if surrounded by \< and \>.

    Cut:
    cut -c1-n        -- cuts from first character to nth character
    cut -b1-n         -- cuts from first bite to nth bite
    cut -f1,2,3 -d'|' -- cuts required no of fields

    VI Editor:
    :wq      -- To save the changes and exit vi editor
    :q!      -- To not to save changes and exit
    h        -- move cursor left one character on current line
     j       -- move cursor down one line
    k        -- move cursor up one line
     l       -- move cursor right one character on current line
    esc      -- to change the insert mode to command mode
    o        -- to start a new line
    i       -- to write from the point you want
    a        -- to write from the last of the line
    x        -- delete the character which is under the cursor
    dd         -- delete the line which the cursor is on
    dw          -- delete the word which the cursor is on
    u        -- undo the last editing change
    /x        -- to search a string(x)
    :%s/x/y/g -- replace x with y
    :10,20s/computor/computer/g -- to replace strings for line through 10 to 20

    Practice Unix Scripts:
    #IF Validations:

1. if [ $? -ne 0 ]

    Then echo "Error Occurred While SCP"


2. if [ $# -ne 1 ]
Then echo "Number of arguments is not equal to 1. You wouldn't have provided any argument or would
have provided more than one argument"
#EXPR command:

    1. expr 256 % 10 will return 6
    2. expr 256 / 10 will return 25

# How to kill the session
$ps -ef|grep sjohn
sjohn 21371 21368 0 00:08:52 pts/7       0:00 -ksh
sjohn 15232 15168 0 07:08:22 pts/2       0:00 -ksh
sjohn 24870 24790 0 07:50:01 pts/17       0:00 -ksh
sjohn 7526 24870 0 08:03:01 pts/17       0:00 grep sreddy
$kill -9 21371

# Variable assign from sql query and creating session parameter file
v1=`sqlplus -s user/pwd@db <<cnt
set heading off
set feedback off
select count(0) from dual;`
echo "[session_name]">test_para.txt
echo '$$v_cnt='$v1 >>test_para.txt

# If the data is present in the below format
MONTGOMERY                AL 36131 Y Y N N N 17 17 0 101 P C Y
MTGY                AL 36131 Y Y Y N N 25 17 0 101 N C N
STATE DEPT INDUSTRIAL RELAT AL 36131 N Y Y N N 20 17 0 101 N C N
MONTGOMERY                AL 36132 Y Y N N N 17 17 0 101 P C Y
MONTG                 AL 36132 Y Y Y N N 23 17 0 101 N C N
MTGY                AL 36132 Y Y Y N N 25 17 0 101 N C N
STATE OF ALABAMA REVENUE DEP AL 36132 Y Y Y N N 21 17 0 101 N C N

for picking the first record based on zip code.
1) create a file b.txt with unique zip code values
36131
36132

2)
while read a
do
grep $a data_file_name|head -1>>postal_unique1.txt
done <b.txt

#Replace 8th field is !=1 with = 1
awk -F"|" '{     ( $8 != "1" ) { $8 = "1" }; { print $1,$2,$3,$4,$5,$6,$7,$8 }}' OFS="|" file_name > tt
#Taking the values from file and making insert scripts.
awk '{printf("insert into machine_node_delete (machine_node_id) values (%s);\n",$1)}' G:/log-
reporting/mid_list.txt > G:/log-reporting/mid_insert.sql
###############################################################################
I have an input file (input.dat):
0 0.24 0.35 0.5 0.6 0.2 0.12
1 0.4 0.2 0.45 0.55 0.3 0.40
2 0.3 0.3 0.4 0.4 0.45 0.5
3 0.2 0.1 0.4 0.5 0.34 0.2
then I use if command :
awk '{if ($2 < 0.35) print $1, 1, $2, $3, $4, $5, $6, $7; else print
$1, 0, $2, $3, $4, $5, $6, $7}' input.dat > output.dat
The output.dat:
0 1 0.35 0.5 0.6 0.2 0.12
1 0 0.2 0.45 0.55 0.3 0.40
2 1 0.3 0.4 0.4 0.45 0.5
3 1 0.1 0.4 0.5 0.34 0.2
and I can repeat the if-command for the next column, so that the final
output:
0 110011
1 110010
2 110000
3 110011
Does anyone know the short way to perform the command?
Command:- awk -v t=0.35 '{ print $1, $2<t, $3<t, $4<t, $5<t, $6<t, $7<t } ' file_name
result :

 0100011
 1010010
 2110000
 3110011

####################################################################################
# File exists check
if [ -e "filename" ]
then
echo "File exist"
else
echo "File does not exist"
fi

# Loops
------
1)
for i in `seq -w 1 100`; do echo "$i";
done

2)
for i in `seq -w 1 100`; do printf "$i\r";
sleep 1;
done
# Finding dining word in data.file and print the succeding 3 lines
sed -ne "/ dining /{n;p;n;p;n;p;}" data.file > output.file

# For compressing the folder temp and creating zipped folder temp1.zip
zip -r temp1 temp

#For searching the string "James" in all sub directories.
 Go to the corresponding directory.
 grep -r James .
(or)
find [dir] -name [files] | xargs grep [text to search for]
ex:- find . -name "*.*" | xargs grep HSBC0400002


# Get common records from more than one file
grep -Ff file1 file2

#Fixed width file record length or for getting record length
awk '{print length () }' file_name.txt|uniq

# If the file has the below data....
File name: a.txt

I,ABCD,XYZ
I,DKKD,4333
U,DFDFD,DFDWAS
I,DFDF,1322
I,DFDF,3
U,232,FSDFDS
1)To print first column contain "I" ?

A)awk -F"," '$1 ~ /I/' a.txt
Print the 2nd column where ever XYZ string is there in the record
A)awk -F"," '/JAMES/ { print $2 }' a.txt

2) file record count by use awk
awk 'END {print NR}' infile
3) count no of non empty records (except blank lines)
 awk 'NF != 0 {++count} END {print count}' list
4) to print first row values with out gap in column format
awk '{ print $1 "\n"}' temp.txt>temp1.txt
5) to print first column of the data (delimiter is comma) in begin block.
awk -f myscript.awk myfile.in
myscript.awk
--------------
BEGIN {
      FS=":"
}
{ print $1 }
6)line not starting with I
sed -e '/^U/d' ims_import_imsexp.txt|wc -l
grep ^u ims_import_imsexp.txt|wc -l

# Find how many number of times the string was repeated in the file
awk -F"," '/lname/{print FNR}' a.txt|tail -1

# give colors for folders.
$ ls --color=tty

# For execution of the previous command.
$ !!

# For execution of the related command
ex:- !cp (For executing the last cp command)

#print list of disks and current capacity (in kilobytes)
$ df -k

#To remove directory along with files
$ rm -rf ( -f option for removing forcefully)

# copy dir.1 (and its contents) to dir.2 ( Means copying the directories)
$ cp -r dir.1 dir.2 (-r option is for doing recursively)

# Print expanded list of all processes ( Means currently running process in the UNIX box)
$ ps -ef

#lists all files in the parent directory
$ ls ../

#For getting record count in the file
$ awk 'END {print NR}' a.txt
or

# For printing non null rows ( skipping empty rows/lines)
$ awk 'NF != 0 {++count} END {print count}' file_name

# For checking the no of fields in a file
$ awk -F"," '{print NF}' a.txt

# assign variable from outside
$ awk -F"," -v t1='James' '{print t1}' dummy_file.txt

# If the 5th field is null the print "US"
$awk -F"|" '{if ($5=="") {$5="US"}; print $0;}' OFS="|" ODS_Lilly_Prescriber_Specialty.dat
Several special shell variables are predefined. Some useful ones are
   $#, $*, $?, and $$.
Arguments can be passed to a shell script. These arguments can be accessed inside the script by using
the shell variables $1, $2,...,$n for positional parameter 1,2,...,n. The filename of the shell script itself is
$0. The number of such arguments is $#. For example, if file do_it is a shell script and it is called by
giving the command

#Removing header record and footer record
tail +2 a.txt|sed -n '$!p'
(or)
tail +2 a.txt|sed '$d'

#Summation of the field 1
$ awk -F"," 'BEGIN {x=0} {x+=$1} END {print x}' a.txt

#For priniting the list of aliases present under your user.
$ alias

# Sort name has comma in between. Remove comma between 92-203 characters.
awk '{ fir=substr($0,1,111)} {srt=substr($0,112,92) }{ lst=substr($0,204,485)}{sub(/,/,"",srt)} {srt=srt" "}
{srt=substr(srt,1,92)} {print fir srt lst}' ims_rxdm.out

#for joining two files (first file has number and second file has data).
$ sed -n -f file1 file2
in this - file1 is script file. The data should be like below
/^0000111/p
/^0000122/p
/^0000128/p
/^0000155/p
/^0000167/p
/^0000186/p

Here what ever the values starting (^) line with that number in the seconde file , it will print it.

#maximum length of the 5th column in the file.
awk -F"|" '{ if (length($5) > max) max = length($5) } END { print max }'
CUSTOMER_PRODUCT_SEGMENT.txt
awk -F"|" '{if ($10=="HF") {$10="101"}; if ($10=="LF") {$10="100"}; print $0;}' OFS="|"
CUSTOMER_PRODUCT_SEGMENT.txt

#####################################################################################
###########
# sum of all digits for the given number
if [ $# -ne 1 ]
then
   echo "Usage: $0 number"
   echo "      I will find sum of all digit for given number"
   echo "      For eg. $0 123, I will print 6 as sum of all digit (1+2+3)"
     exit 1
fi

n=$1
sum=0
sd=0
while [ $n -gt 0 ]
do
   sd=`expr $n % 10`
   sum=`expr $sum + $sd`
   n=`expr $n / 10`
done
   echo "Sum of digit for numner is $sum"
#####################################################################################
##########
# number of files of the given type
count=1
for file in `ls -1 *.sh`
do
   echo "$count: $file" >> $mnu0
   count=`expr $count + 1`
done
#####################################################################################
##########
# to check the given number is the range
echo "Please enter a list of numbers between 1 and 100. "
read NUMBERS

for NUM in $NUMBERS
do
         if [ "$NUM" -lt 1 ] || [ "$NUM" -gt 100 ]; then
                  echo "Invalid Number ($NUM) - Must be between 1 and 100!"
         else
                  echo "$NUM is valid."
         fi
done
#####################################################################################
##########
# Validate the password
valid_password="secret"
echo "Please enter the password..."
read PASSWORD
if [ "$PASSWORD" -eq "$valid_password" ];then
      echo "You have access!"
else
         echo "ACCESS DENIED!"
fi
#####################################################################################
##########
# Greeting
temph=`date | cut -c12-13`
dat=`date +"%A %d in %B of %Y (%r)"`
if [ $temph -lt 12 ]
then
   mess="Good Morning $LOGNAME, Have nice day!"
fi
if [ $temph -gt 12 -a $temph -le 16 ]
then
   mess="Good Afternoon $LOGNAME"
fi
if [ $temph -gt 16 -a $temph -le 18 ]
then
   mess="Good Evening $LOGNAME"
fi
#####################################################################################
##########
MAX_NO=0
echo -n "Enter Number between (5 to 9) : "
read MAX_NO
if ! [ $MAX_NO -ge 5 -a $MAX_NO -le 9 ] ; then
   echo "I ask to enter number between 5 and 9, Okay"
   exit 1
fi
clear
for (( i=1; i<=MAX_NO; i++ ))
do
   for (( s=MAX_NO; s>=i; s-- ))
   do
      echo -n " "
   done
   for ((j=1; j<=i; j++))
   do
    echo -n " ."
   done
   echo ""
done
#####################################################################################
##########
awk -F"|" '{arr[$3]+=$4} END {for (i in arr) {print i, arr[i]}}' abc.txt
awk -F"|" '{arr[$3]++} END {for (i in arr) {print i, arr[i]}}' abc.txt
sort file | uniq -c
#####################################################################################
##########
#3sum, avg, min, max
sum=0
min=10000
max=0
n=5
i=0
file="students.txt"
cat $file | while read line; do
set line
grade=$3
sum='expr $sum + $grade'
if [ $grade -gt $max ]; then
max=$grade
fi
if [ $grade -lt $min ]; then
min=$grade
fi
i='expr $i + 1'
if [ $i -ge $n ]; then
break
fi
done
average='expr $sum / $i'
echo "Max: $max Min: $min Avg: $average"
#####################################################################################
############
Sed: (stream editor) is a non interactive text editor which is considerably faster than interactive text
editors. The command line syntax for sed is:
sed [ options …][command][files …]
The different permissible options are:
-f file the file contains editing commands
-n suppress automatic printing of lines
-e the script is made up of one or more commands
-r means use extended regexps
sed -r -n -e '/PATTERN/,${p}' to print the line from the first match till end
awk '/30/{i++}i' file

# CUT, SORT, UNIQ, CMP,Split Commands
cmp command compares the two files
cut command selects a list of columns or fields from one or more files.
 uniq command removes duplicate adjacent lines from sorted file while sending one copy of each second
file
 sort dupsls.txt|uniq to display uniq records from the file.
 cut -f2,5 -d "|" psales.txt # will display the data from 2 and 5 fields with | delemeter from file.
   awk -s '{FS="|"}''{print $2,$5;}' dupsls.txt # will display the data from 2 and 5 fields with out delemeter
from file.
cut -f2,5 -d "|" psales.txt|sort|uniq >New_Psales.txt # command will cut 2-5 fields from psales.txt and
sort data and write unique records to a new file New_Psales.txt
 split -l 100 psales.txt # will split the file into multiple files with 100 lines in each file.
 split -b 40k psales.txt # will split the file into multiple files with filesize as 40k each.
# Translate Examples:
tr -d '\r' < input_dos_file.txt > output_unix_file.txt
echo @abcd1234 | tr -cs '[:lower:]''[:upper:]' '[\n*]' | sed '1d'`
echo AbcDeF12Gh345 | tr -cd '[:lower:][:digit:]' bce12h345
echo AbcDeF12Gh345 | tr -d '[:lower:][:digit:]' ADFG

# Sort on Required Fields:
sort -t'|' -k4.1 bbc.txt sort -r -t'|' -k4.1 bbc.txt

Following are some sed examples
1. Print all lines from the file phonebook that end with 76 :
sed -n '/76$/p' phonebook
Notice /76$/ tells sed to search for lines which end with 76. Dollar sign means end of the line. The -n is
needed and p means print.
2. Print all lines from the file phonebook that start with Su :
sed -n '/^Su/p' phonebook
Notice /^Su/ tells sed to search for lines that start with SU. Carret ^ sign means beginning of the line.
3. Print all lines from the file phonebook that contain 76 :
sed -n '/76/p' phonebook
Notice /76/ tells sed to search for lines which contains 76. This has the
same effects as the Unix grep, fgrep and egrep commands.
4. Print all lines from the file phonebook that showing nonprinting characters (such as tab)
sed -n 'l' phonebook
Notice between the two single quotes is the letter l not number 1.
5. Delete 4 lines from the file phonebook:
sed '4d' phonebook
Notice d is for delete.
6. Delete all lines from the file phonebook that start with Su :
sed '/^Su/d' phonebook

7. Delete all lines that end with 76 from the file phonebook:
sed '/76$/d' phonebook
8. Delete all lines that contain 76 from the file phonebook::
sed '/76/d' phonebook
10. Delete all lines that contain Billy or billy from the file phonebook:
sed '/[Bb]illy/d' phonebook
Notice the rgular expression [Bb] first takes B (hence it will be Billy) and next takes b (that makes billy).
11. Replace the first 4 characters of each line of the file phonebook by MOON
sed 's/..../MOON/' phonebook
Notice s stands for substitute, and period . is the regular
expression which means any character. Thus the expression s/..../MOON/ means search for the first 4
characters and substitute with MOON.
12. Replace the last 4 characters of each line of the file phonebook by MOON
sed 's/....$/MOON/' phonebook
13. Delete the first 4 characters of each line of phonebook
sed 's/....//' phonebook
Notice the subsitute pattern // is null, means replace it by nothing.
14. Delete the last 4 characters of each line of the file phonebook
sed 's/....$//' phonebook
15. Replace the first r found in all lines that contain 98 with R of
the file phonebook.
sed '/98/s/r/R/' phonebook
Notice only Barbara Swingle has 98 and r. Thus the name will be: BaRbara Swingle.
16. Replace all r found in any lines that contains 98 with R of the file phonebook.
sed '/98/s/r/R/g' phonebook
Notice the g means global. Only Barbara Swingle has 98 and r. Thus the name will be:
BaRbaRa Swingle.
17. Print all lines between the line that starts with BEGIN and another line, which starts with END from
file called awk5.
sed -n '/^BEGIN/,/^END/p' awk5
Notice the , between /BEGIN/ and /END/

				
DOCUMENT INFO
Shared By:
Stats:
views:70
posted:4/2/2011
language:English
pages:11