Introduction to the School of Earth Sciences UNIX computer network
Kevin Keay February 8 2008
Outline
Course materials What is UNIX? Our UNIX network A brief discussion of why we use UNIX A taste of UNIX A basic overview A typical UNIX task The University of Surrey, UNIX Tutorial for Beginners: Selected topics An Introductory Tutorial on UNIX (a lab session)
Course materials
Handout:
An Introductory Tutorial on UNIX
which is also at:
http://www.earthsci.unimelb.edu.au/~kevin/UNIX_Course/An_
Intro_Tute_on_UNIX_Feb-2008.pdf
Notes: at:
Introduction to the School of Earth Sciences UNIX computer network
http://www.earthsci.unimelb.edu.au/~kevin/UNIX_Course/Intr o_to_SES_UNIX_computer_network.pdf
A useful online resource from the University of Surrey, UNIX Tutorial for Beginners, is available at: http://www.ee.surrey.ac.uk/Teaching/Unix/
What is UNIX?
UNIX is a computer operating system Well suited to multi-tasking and multiple users Powerful command-line and scripting capabilities Graphic user interface: X Windows Easily accessed from Microsoft Windows (PC) or X Darwin (iMac) as well as a native UNIX machine Important: UNIX is case-sensitive e.g. nedit is different from NEDIT or Nedit
What is UNIX?
(2)
UNIX was developed initially by Bell Labs circa 1970 and has grown into an open software project with many world-wide collaborators. A history is given at: http://www.bell-labs.com/history/unix/ There are many variants e.g. Linux, Mac OS X. Cygwin for Windows is a UNIX emulation i.e. looks and feels just like UNIX.
What is UNIX?
(3)
After three decades of use, the UNIX computer operating system from Bell Labs is still regarded as one of the most powerful, versatile, and flexible operating systems (OS) in the computer world. Its popularity is due to many factors, including its ability to run on a wide variety of machines, from PCs to supercomputers, and its portability - all of which led to its adoption by many manufacturers.
Our UNIX network
atlas, mawson, orthus, rossby cove, gulf, gyre, tide, wave vislab01, vislab02, vislab03, vislab04 (4) OSF1 OS: zeus (OS means operating system). Group 3 are the newest and fastest machines. Group 2 are recommended for most uses. Apart from atlas and orthus, the group 1 machines are relatively slow. Group 4 is for specialised uses e.g. GCM work. So, normally you would select a Linux machine in group 2 or 3 unless you need to use the Solaris OS. Also, see section 1.1 of the notes
(1) Solaris OS: (2) Red Hat Linux OS: (3) Debian Linux OS:
Introduction to the School of Earth Sciences UNIX computer network.
A brief discussion of why we use UNIX
UNIX machines are generally better at multi-tasking than MS Windows Very stable platform from a user standpoint: difficult to crash! Compilers (Fortran, C) allow the user to development their own software or implement programs written by others There are many sophisticated commercial packages available e.g. Matlab
A brief discussion of why we use UNIX
(2)
Scripts can be used to automate complex sequences of tasks: leads to efficiency and reduction of user errors Command-line arguments add flexibility and are very useful in scripts e.g. C-shell scripts There is a global resource of software designed to work under UNIX, much of it is freely available: you don’t have to develop everything from scratch
A taste of UNIX
Imagine that you have connected to a UNIX machine e.g. cove, via ssh from a Cygwin xterm window: ssh –X cove You will find yourself located in your UNIX home folder e.g. /home/kevin
A taste of UNIX (2) Change to another folder cd /work9/kevin/Project1 /work9 is another disk and the user kevin may create folders and files within it Check that you are in the correct location (optional!) pwd /work9/kevin/Project1
A taste of UNIX (3)
List the contents (files) of this folder ls
pmsl.cmp sortdata.f Test.dat test2.dat X.dat
List the files with ‘.dat’ as part of the filename ls *.dat Test.dat test2.dat X.dat
A taste of UNIX (4)
Edit the Fortran source file sortdata.f (we are making changes to the existing file) nedit sortdata.f
A taste of UNIX (5)
Compile the Fortran source program to create an ‘executable’ i.e. a piece of software g77 –o sortdata sortdata.f Here the –o option expects an argument, in this case the filename of the executable (an option could be standalone e.g. –j –s) List the files in the folder (sortdata is a new file) ls pmsl.cmp sortdata sortdata.f Test.dat test2.dat X.dat
A taste of UNIX (6)
Run the executable (software) that you designed in Fortran sortdata You might design your program to create an output data file or the results of a computation, even an image
A taste of UNIX (7)
Create a plot of a pressure file conmap –k iconkk –B pmsl.cmp < icon The items with ‘-’ are called options and control the program called conmap. The –k option has an argument – a file called iconkk which contains some additional parameters to control the program Additional control comes from the instruction file called icon The output of this command is a file called gmeta that can be converted to a PNG image
A taste of UNIX (7)
A basic overview
When you connect to a remote UNIX machine on our network you will be working in the C-shell (actually tcsh which is aliased to csh) C-shell scripts will begin with # or #!/bin/csh -f There are other kinds of script e.g. sh (Bourne shell – bash under Cygwin is a relative), perl After connection you will be located in: /home/user where ‘user’ is your UNIX username e.g. /home/kevin Remember: UNIX is case-sensitive
A basic overview (2)
Under MS Windows files and folders (directories) are usually relative to the C drive i.e. c: and under UNIX are relative to / Also \ is used to denote the file tree structure under Windows and / under UNIX Hence: c:\home\kevin\test.dir\prog.f under Windows would be: /home/kevin/test.dir/prog.f under UNIX You will be working under /home or /worknn e.g. /home/kevin, /work5/kevin, /work18/kevin
A basic overview (3)
ls is used to list files in a folder e.g. ls readme.txt ls –l pmsl.*.dat Note that the ls command has many options. See: man ls for more information
A basic overview (4)
mkdir is used to create folders (directories) e.g. mkdir test.dir rmdir is used to remove a folder e.g. rmdir test.dir Note: The folder must be empty i.e. use rm first, then rmdir chmod is used to change file access permissions e.g. chmod 755 readcmp
A basic overview (5)
rm is used to remove (delete) files e.g. rm test.dat By default rm will remove without warning. Your initial setup will alias rm such that rm == rm –i i.e. you will be prompted to continue with the removal Prefixing with a \ will override the alias e.g. \rm –rf */* will delete all of your files starting with the current folder – BEWARE!
A basic overview (6)
cp is used to copy files to other locations e.g. cp readme.txt /work18/kevin/src mv is used to rename files e.g. mv readme.txt Readme.txt Remember: UNIX is case-sensitive pwd will tell you where you are i.e. your current location (folder)
A basic overview (7)
cd is used to change folders (directories) Assume you are in: /home/kevin cd test.dir will change the folder (directory) to test.dir i.e. you are now in: /home/kevin/test.dir cd .. will place you in the folder one level above i.e. in this case: /home/kevin cd ~ will always take you ‘home’ i.e. to: /home/user
A basic overview (8)
cd /home/kevin/src will place you in the folder called src cd /work18/kevin/src/readgrib.dir will place you in the folder readgrib.dir under /work18/kevin/src Combine .. and / to change folders e.g. cd ../test2.dir
A basic overview (9)
Wildcard characters *,?,- and , in conjunction with various brackets {} and [] are very useful in commands to represent complex sets of filenames e.g. cp pmsl.*.dat /work18/kevin/DATA This will copy all files of this format where the * stands for anything in that part of the filename.
A basic overview (10)
? represents a single character e.g.
cp pmsl.{19[6-9]?,2000,2001}.dat /work18/kevin/DATA
This will copy: pmsl.1960.dat,pmsl.1961.dat,…,
pmsl.1999.dat,pmsl.2000.dat, pmsl.2001.dat i.e. {19[6-9]?,2000,2001} is expanded to: 1960,1961,1962,…,1999,2000,2001
We could also use: pmsl.{19[6-9]?,200[0,1]}.dat
A basic overview (11)
For text editing use nedit on atlas or orthus and nedit or emacs on the Linux machines e.g. nedit prog.f To transfer files to and from a PC you can use the utility SSH Secure File Transfer Client on the Windows desktop
You can also use scp under Cygwin – see Appendix A of the Notes
A basic overview (12)
Some documentation is available with man or info, however it is often cryptic e.g. man csh Often a Google web search will find more helpful information, examples and tutorials
A basic overview
(13) Summary
The basic UNIX commands to understand are:
ls cd mkdir rmdir rm chmod cp mv pwd csh
Learn how to use wildcards:
* ? - , [] {}
Ensure that you can use a text editor: nedit (Solaris OS) nedit emacs (Linux) SSH Secure File Transfer Client (Windows) is useful for transferring files from a PC to a remote UNIX machine Also see: scp
A typical UNIX task
Task: Prepare average geographical maps of mean sea level pressure (MSLP) for the months of Jan and Aug using NCEP Reanalysis data and plot the results The data is available in a widely-used format called NetCDF from the NOAA web site – this site has interactive data selection features I manually downloaded the relevant files using a web browser (Mozilla) to my UNIX account e.g. /work9/kevin/mean_MSLP.dir
A typical UNIX task
(2)
I have written a Fortran program (read_nc2cmp) to decode the NetCDF files to our internal conmap format This program was based on an example from NCAR so I didn’t have to begin from scratch The conmap files are then input to my Fortran program statconmap which produces the average maps Note: The average is performed at each gridpoint of the map for the set of desired maps
A typical UNIX task
(3)
The average maps (one for Jan, the other for Aug) may be plotted with various software e.g. NCAR Graphics, GrADS, Matlab One method that is useful in scripts is: (1) run conmap (NCAR Graphics) on an average map; output is a file called gmeta (2) Translate gmeta to a Postscript file using the utility g2ps (3) Use the utility convert to transform the Postscript file to a PNG file – this may be pasted into a Word document
A typical UNIX task
(4)
In a UNIX C-shell window you can type commands directly e.g.
read_nc2cmp MSLP.*.Jan.nc out.cmp
statconmap out.cmp ave.Jan.cmp sd.cmp var.cmp conmap –G ave.Jan.cmp
You can also place these commands in a C-shell script
A typical UNIX task (5) #!/bin/csh –f # Required first line of a C-shell script or simply # set mons = (Jan Aug) # Array of months @ nmons = $#mons # No. of months, in this case it will be 2 @i=1 # @ is used to define numbers as opposed to text (set) while ($i <= $nmons) # Perform loop (to end) while i <= 2 set m = $mons[$i] # m is mons(i) i.e. mons(1) = Jan => m = Jan # Decode NetCDF files for this month (m) # Note: out.cmp contains multiple maps read_nc2cmp MSLP.*.$m.nc out.cmp # Perform average of multiple conmap file out.cmp # Note: Outputs sd.cmp and var.cmp are not used but need to be specified statconmap out.cmp ave.$m.cmp sd.cmp var.cmp # Create global geographical plot of average map e.g. ave.Jan.cmp conmap –G ave.$m.cmp # Output is gmeta # Convert gmeta file to Postscript; creates g.ps (a hardwired output) g2ps gmeta # Convert Postscript file to PNG e.g. MSLP.Jan.png convert –trim –density 120 g.ps MSLP.$m.png # @ i ++ # i -> i + 1 end exit
A typical UNIX task
(6)
We won’t look in detail at scripts in this introduction but will re-visit them later in the semester Remember: Practice makes perfect! Experiment … play with the commands and see if things make sense
The University of Surrey, UNIX Tutorial for Beginners: Selected topics
See: http://www.ee.surrey.ac.uk/Teaching/Unix/ Note: Tutorial 7 is a bit advanced for this introduction
An Introductory Tutorial on UNIX
Refer to the handout Also see:
Appendix C An Introductory Tutorial on UNIX
of the notes
An Introductory Tutorial on UNIX Initial setup of a UNIX account
(2)
Connect to a remote UNIX machine e.g.: ssh –X cove.earthsci.unimelb.edu.au After logging on:
% cp /home/kevin/UNIX_Course/zunix_setup.zip . - Note the dot % unzip zunix_setup.zip % ls –l .cshrc.* % exit
An Introductory Tutorial on UNIX
(3)
Owing to recent changes to atlas use a Linux machine like cove, tide or gyre: e.g. ssh –X cove.earthsci.unimelb.edu.au In step 8 you will need to convert the SUN conmap file to a PC version with binswap i.e. binswap –c pmsl.ncep.1996060106.cmp{,} Note: {,} replaces pmsl.ncep.1996060106.cmp as the output. You can use a different output name if you wish e.g. binswap –c pmsl.ncep.1996060106.cmp pmsl.cmp