Document Sample
spider Powered By Docstoc
					    Spider Data Analysis and Other Useful Info
                  and Procedures
                            Last updated 06/2006 by JB

Table of Content

2-3     –   Commonly Used UNIX Commands

4       –   Standard Calculations we often perform

5-7     –   Summary of Codes we use

8       –   Standard Structure of Project Directory

9-21    –   Description of Spider codes

            9-13    -- for UNTILTED micrographs

            14-16   -- for TILTED micrographs

            17      -- for Defocus Pairs

            18-19   -- for CTF determination

            20      -- for Common Lines

            21      -- for Reference-Free Alignment and Classification

            22      -- other codes

23-25   –   Description of Refinement

Commonly used UNIX commands
   To get more information about any of the UNIX commands type
   >man command_name

   ls -- lists all the files in a directory
   ls –l -- lists all the files in a directory, their sizes, and associated permissions
   mkdir ser -- used to create a directory named “ser”
   rmdir ser -- used to remove empty directory named “ser”
   rm -fr ser -- used to remove non-empty directory named “ser”
   rm filename -- used to remove a file, you‟ll be asked for confirmation
   rm –f filename -- used to remove a file without further verification (NOTE: this operation is
   irreversible thus be careful when using it)
   cp filename destination/new_filename -- used to copy a file from one location to another
   cp –fr directory destination/new_directory -- used to copy a directory from one location to another
   mv filename destination/new_filename -- used to move a file from one location to another (use mv
   filename destination/. if you want to preserve file name)
   mv –f directory destination/new_directory -- used to move a directory from one location to another
   more filename -- used to display (not edit) the content of file in the terminal window
   tail -100 filename -- used to display in the terminal window last 100 lines of the file
   nedit or jot filename -- use to display/edit file in a separate window
   vi filename -- used to edit file in terminal window (not very convenient, check details with man)
   pico filename -- used to edit file in terminal window (more convenient than vi)

   pwd -- shows your current location on computer
   whoami -- shows under whose name you are currently logged in
   uname –a -- displays system‟s name and operating system
   df -- shows used/available/total space on disk
   du directory – shows how much space specified directory occupies
   top -- used to check the processes which currently run on the computer
   kill process_ID -- used to kill process that currently runs on computer (process_ID is shown by top
   chmod 700 filename/s -- used to change file/s permissions (first digit – your permissions; second
   digit – group permissions; third digit – outside users permissions. Numbers mean the following: 1-
   execute, 2-write, 4-read; for combination use sum of numbers. For ex., 3 gives execute and write
   chmod -R 750 directory -- used to change permissions for all files within specified directory
   chown owner_name filename/s – used to change owner of the file/s
   chgrp group_name filename/s – used to change to which group file/s belong
   rl (or telnet or ssh) chispa -- used to log in to another computer from your computer (rl – not secure,
   don‟t run jobs through such connection – will slow them significantly; telnet – more secure, won‟t slow
   jobs; ssh – secure, won‟t slow jobs)

scp filename user@remotehost:distination_foder -- used to copy files to remote computer (when
working within our computer cluster use usual cp command)
gunzip/gzip -- used to compress/decompress files/directories add/remove extension .gz
tar xvf name.tar -- used to un-tar (decompress) file/directory with extension .tar
tar cvf name.tar directory – used to compress specified directory, will name output file name.tar
lp –dlj44 filename -- used to print a file on our black and white printer (can be used from terminal
window as well as from any application such as Netscape)
lp –damicolor filename – used to print a file on the color printer downstairs

dmf put filename/filenames/directory destination -- used to put files to the DMF archiver
dmf get filename/directory destination -- used to get files from DMF
dmf tar c destination/name.tar directory -- used to compress directory and put it to DMF with the
specified name
dmf tar x destination/name.tar – used to decompress content of name.tar from DMF to current

Standard calculations we often perform

A/pixel calculation

         Microscope magnification --- 66,000 (CM 200, ice data) and 60,000 (CM 120, neg. stain data)
         Scanner resolution 7 mkm/pixel
         Decimation during cpfromraw.spi 2 times (ice data) and 3 times (negative stain data)

         Thus, for ice
                         -6                                   -10
         A/pixel = 7*10 m/pixel*2/66,000 = 2.12*10 m/pixel = 2.12 A/pixel
         we use 2.055 A/pixel (why?? it‟s 3% less than 2.12 A/pixel)

         For negative stain
                         -6                                  -10
         A/pixel = 7*10 m/pixel*3/60,000 = 3.5*10                  m/pixel = 3.5 A/pixel

Frequency for use in Spider filtering commands

         Frequency for filtering should be provided in 1/pixel units.
         We know A/pixel for our images (see above). Also, we know desired resolution to which we want to
         filter, thus

         1/pixel = [A/pixel] / [desired resolution in A]

Number of Voxels corresponding to given Molecular Weight (MW)
         Average protein density = 0.83 Da/A

                                          [MW in Da]
         #voxels    =         --------------------------------------------     = # pixel
                                             3                           3
                              [0.83 Da/A ] [A/pixel, e.g. 2.055]

Rotating 3D structures in Web

         X         +    counterclockwise
                   –    clockwise

         Y         +    top rotates into the screen away from us
                   –    top rotates out of the screen towards us

         Z         +    right side rotates into the screen away from us
                   –    right side rotates out of the screen towards us

Correspondence between 3D images and their projections displayed in Web

When generating projections images we create file which stores 3 Euler angles for each projection
col1 – a        col2 – b          col3 – c

When displaying 3D image in Web to put it into the same orientation as a particular projection use:
X = c – 90       Y = b – 90      Z=a

Summary of codes to use (in order)

Current version of all codes mentioned here can be found at /nfs/cuatro/cuatro2/usr/programs/
Extension name „spi‟ should be substituted by your project‟s extension.

All command files are written assuming that you‟ll create directory commands in your project
directory and copy all of the command files in there. Use extension.script to change all the
Commands are to be run from the command line in the top most directory of your project, unless
otherwise specified.

All Spider programs are run by typing >spider spi
‘en’ or ‘en d’ can be used to stop any spider operation. „en‟ preserves the generated results file,
while „en d‟ deletes the results file.

For each project you have please create a document file prjname.doc in the
/cuatro2/usr/programs/projects directory and use it for making notes on all the steps you do.

General steps for untilted particles:
create directory micros, put all .raw and .tif images in there
microsdoc.script run this from micros directory it will create file microsdoc.spi in the directory
cpfromraw.spi – takes ***.raw or ****.raw as input files
automatically select particles or handselect particles in Web (commands->Pixel) write them in the
files handpick****.spi
dmf put *.tif *.raw project_directory/. – check that files were successfully saved to dmf and after that
delete them
create characteristic noise window: (i)noise.spi
view particles in Web, choose bad ones (commands->categorize), save in file handsort.spi
reseq.spi – eliminates bad particles and lists particles statistics to particles_corr.spi
filter.spi – filters particles
create directory vol, copy there reference volume vol001.spi
volpdsh.spi (if needed)– inter/extra-polates volume to correct A/pixel, centers it and puts into
window of specified size
refprj.spi – makes 83 projections out of given reference volume
apmq.spi – finds alignment parameters for each particle with respect to 83 reference projections
and their mirrors
sort.spi – in sort directory creates files containing info about particles‟ numbers for each

To do reference-free alignment and process plus and minus groups
endline.spi – creates document files with the number of images belonging to each of the plus
and minus projections, these files will be used by plotp.spi and plotm.spi
plotp.spi, plotm.spi – plot distribution of particles through different views (projections)
shrm.spi (calls p_rotate.spi) – shifts, rotates and mirrors particles in accordance with
parameters found by apmq.spi, writes results to ser02fa
angles.spi (calls p_angles.spi) – finds three Euler angles for each particle
bpcg.spi – creates volume using filtered and aligned by shrm.spi particles, using angles of the
reference volume (e.g. angles.spi file)

rmreffp.spi, rmreffm.spi – run ten rounds of ap sr on particles in each projection group
(separately for plus and minus groups)
orgapsr.spi – for each plus and minus group runs one round of ap sr on ten ap sr averages from
rmreffp.spi and rmreffm.spi

To do reference-free alignment and process plus and minus groups
together, e.g. combined:
ser02fm.spi – copies all particles from ser02f mirroring the ones which belong to „-„ groups
bpcgcomb.spi – creates volume using 83 ap sr avarages

Alignment for tilted pairs:
newwcut_t.spi and newwcut_u.spi – analogous to newwcut.spi for untilted micrographs
reseqelim.spi – moves hand rejected particles from directories ser and dser to directory reject
reseqptdoc.spi – copies good particles into directories ser02 and dser02
filter.spi – filters particles to specified resolution
apsr.spi – finds best alignment for untilted particles with respect to each other
shr02.spi – rotates and shifts untilted particles according to results of apsr.spi
hierarchy.spi – runs hierarchical clustering on particles from untilted micrographs
hierarmerge.spi – merges groups from hierarchy.prj which you want to consider together
crmicnum.spi – creates file containing particles and micrograph #s to which they belong for each
of the groups from hierarchy.
createang.spi – evaluates angles for tilted particles based on ap sr for untilted particles and
angular shift between two micrographs
pjcc_gr1.spi and pjcc_gr2.spi – reconstructs volume based on random conical tilt procedure

Defocus Pairs
lowdefcorr_smooth.spi – correlates close to focus and far from focus micrographs in pairs
if computer does only one loop and crashes for unknown reason try using lowdefcor_wstops.spi
lowdefcorr2.spi – finds coordinates of particles for close to focus micrographs
lowdefcorr2clean.spi – eliminates particles which are outside of micrograph range
newwcut_dfp.spi – cuts particles from close to focus micrographs

CTF determination (can be done anytime after newmicro.spi)
power.spi (calls power_p1.spi, power_p2.spi) – finds 1-dim displayable ctf for each micrograph
ctfminima.spi – finds minimum values in each micrograph‟s 1-dim ctf from power.spi
ctfnonoise.spi – removes noise from ctf, outcome is 1-dim file similar to that of power.spi but
with noise removed
defgrps_cr.spi – creates sample file defgrps.spi which you‟ll be updating
use Web to determine defocus values (Commands -> CTF from doc file; doc file to use
noiserem**.spi) record results into defgrps.spi, then in Exel group micrographs into appropriate
number of defocus groups (defocus spread <1000A, particles per group >500, better 1000) name
results file defgrps.txt
defgrps_xlx.spi – extracts info from defgrps.txt and writes it into defgrps_sorted.spi

defgrstat.spi – creates file defgrpsstat.spi, which contains number of particles and average
defocus for each defocus group
powerspec.spi – calculates weighted power spectrum for each micrograph, and final 1-D
displayable power spectrum for given dataset

Common lines:
create directory op, copy there and modify apsrselect01.spi, create in there directory prj and
copy there projections with which common lines will be working
op.spi / opcomb.spi – finds mutual orientation of the projections by common lines method
nedit rangles02.spi – choose set of angles with minimal error value or with stabilized error value
bpcg.spi – reconstructs 3D volume using angles found by op.spi

Running refinement on cluster
create directory refine
copy there the following files from the directory /nfs/cuatro/cuatro2/usr/programs/refine:
ali3dapmq.spi, ali3daprq.spi, ali3dcatchup.spi, apmqrange001.spi, angdist.spi, moveavg.spi,
rangesize001.spi, refine031103.spi, source.spi, spider5mp, stack.spi, stepsize001.spi, vol001.spi
(your reference volume)

update refine031103.spi with the correct number of particles, defocus groups, etc.
update apmqrange001.spi with correct parameters
defgrpdoc.spi – creates files defgrpdoc0**.spi in the refine directory which contain numbers of
particles belonging to each defocus group
order_select.spi – creates file order_select001.spi in directory refine (same as defgrps2.spi, but
reversed order of columns)
stack.spi – run inside refine directory, it creates file data.spi which contains stack of all particles
from ser02fa directory
dmf tar c refine.tar refine - dmf the whole refine directory
ssh yellowtail - login to yellowtail – SGI cluster, create command file

The following commands are used on yellowtail:
qsub rsi – submit the refinement to cluster
qdel jobnumber – deletes the job from cluster
qstat (or qstat | grep usrname)– shows jobs submitted or currently running on the cluster

Archiving volumes

The volumes folder is stored at /blingbling/volumes/your_project_name and is used for storing
“landmark” volumes throughout the course of a project. It is vital that important volumes be
added to this directory to make them readily available for later access. If it is the first time a
volume is going to be deposited for a new project, make a new directory for the project and copy
the README_template file from /blingbling/volumes/ into the new directory. For each volume
that is copied into your project directory, update the README file with requested information.

Standard structure of project directory

                    _________apsrtest ap mq, ap sr tests (possibly also common lines) on some volume
                    |________command all command files for project
                    |________dfpmicros same as micros, but for defocus pairs
                    |________dfpower same as power, but for defocus pairs
                    |________dser decimated (4 times) originally cut particles
                    |________dser02 cleaned decimated particles
                    |________micros micrographs + particles info files
                    |________power CTF functions of microcgraphs
                    |                           ____confirm test on final volume
                    |________refine_______| refinement directory
project ____________|                           |____results results of refinement
                    |________refprj projections of the reference volume
                    |________ser originally cut particles
                    |________ser02 cleaned cut particles
                    |________ser02f filtered ser02
                    |________ser02fa aligned according to apmq results ser02f
                    |________ser02fmr particles from minus groups of ser02f were mirrored
                    |________sort classification of particles into projection groups + ap sr results
                    |________vol reference volume + some intermediate volumes

All codes can be found in /nfs/cuatro/cuatro2/usr/programs/
To start Spider type
> spider your_extension
When inside Spider to end it type
>en d (will not save results file)
>en (will save results file

Create your project directory and copy all the necessary codes into commands directory inside
your project directory.
“spi” stand for project extension – substitute accordingly. If you want to change extensions of
many files use extension.script (you must copy it to the directory where you changing
>extension.script old_ext new_ext

Desctiption of individual Spider procedures (presented
in order we use them)

Inside Spider codes all modification that you need to make are
line (majority of codes) unless otherwise stated

For UNTILTED micrographs

input files: micros/***.tif or micros/****.tif, identify.script (will be automatically accessed at its
location on cuatro2), micros/microsdoc.spi (if it already exists)
output files: micros/microsdoc.spi
description: runs identify.script which finds dimensions (length & width) of plane 0 in all .tif files
present in the directory from which you run microsdoc.script. Then writes this information to
microsdoc.spi; if microsdoc.spi already exitst will add info at the end of file
important: copy microsdoc.script to the directory where you have .tif files and run it from there

input files: micros/***.raw or micros/****.raw, micros/microsdoc.spi
output files: micros/****.spi
description: converts files from raw to Spider format. Decimates negative stain images 3 times;
decimates ice images 2 times

input files: micro/microsdoc.spi, micro/****.spi
output files: micros/d4****.spi for negative stain images; micros/i****.spi,micros/d4i****.spi,
micros/d7i****.spi for ice images
description: decimates output Spider files from cpfromraw.spi (micros/****.spi) by 4 for negative
stain and by for 4 and 7 for ice

selecting particles
automatically: see manual for Automatic Particle Selection and use our IDL code
manually: manual particle selection is done in WEB: graphical interface, which is part of Spider
1.       Open WEB by typing >web4 spi &
2.       Adjust contrast (OPTIONS -> Contrast), foreground color (OPTIONS -> Color) and cursor
type if needed
2.       Display the micrograph by going to COMMANDS -> Image
3.       To pick particles, go to COMMANDS -> Pixel: enter output document file name, standard
is handpick****
4.       Particle picking is accomplished by clicking on the particles on the screen (try to click on
the middle of the particle as precisely as possible), remember that you can change cursor type. If
you accidentally select something wrong you‟ll have a chance to remove it while cleaning
selected particles

storing to DMF
purpose: to save space we do not store *.tif and *.raw files on our local computers, we store
them on DMF archiver and delete from local computers as soon as they are no longer
immediately needed
what to do:
create your project directory on DMF:
  >mkdir project_directory
store particles on DMF:
  >dmf put *.tif *.raw destination/.
make sure files were successfully stored on DMF (go to DMF, check file sizes by ls –l), after that
delete them

creating noise window
purpose: create image which contains only noise, it will be used to normalize all particles (see
what to do: go to WEB: >web4 spi &
1.      Display one of the un-decimated micrographs (micros/****.spi – for negative stain,
micros/i****.spi – for ice)
2.      Go to EDIT -> Window Last File -> Fixed Size. Enter the window size: it should be
approximately 50% larger than the size of your particles (check different values until you find the
best one) and the name of noise-file, noise.spi – for negative stain and inoise.spi for ice images
3.      The noise window should be selected by clicking on the characteristic portion of the
micrograph without any particles.

input files: micros/handpick****.spi, micros/****.spi
output files: ser/ser*****.spi, dser/ser*****.spi, micros/particles.spi
description: cuts particles with center placed at coordinates provided by handpick****.spi,
adjusts their histogram to that of the noise window (see previous step); generates file with
statistics about how many particles were selected from each micrograph: particles.spi
important: particles must have been selected carefully so that the coordinates provided by
handpick****.spi are close to center of particles: when you cut particles the whole particle must be
cut without portions missing; if this problem does occur try increasing the window size for the
particles. NOTE, size of noise window does not necessarily have to be equal to size of the
particle window.

identifying bad particles
purpose: eliminate obviously bad particles which were accidentally selected, thus clean the data
set for future data processing
what to do:
1.       In WEB go to COMMANDS -> Categorize -> Sequential Montage and display particles
from dser directory (usually 500 at a time), choose 1 category, name results file
2.       You can reduce the size of particles by some factor by going to OPTIONS -> Image, you
need to redisplay particles for the decimation to take effect

3.     Click on the bad particles, it does not matter in which order you click on them. After you
are done with the subset, clear the screen and display next portion (see step 1)

input files: micros/particles.spi, micros/handsort.spi, ser/ser*****.spi, dser/ser*****.spi
output files: ser02/ser*****.spi, dser02/ser*****, micros/particles_corr.spi, micros/particles2.spi
description: eliminates bad particles (according to handsort.spi) and stores good particles to
ser02 and dser02 directories. Writes out correspondence between particles in ser and ser02
directories to particles_corr.spi. Calculates statistics about how many good particles were
selected from each micrograph: particles2.spi

input files: ser02/ser*****.spi
output files: ser02f/ser*****.spi
description: filters images to desired resolution, to minimize noise contribution in data
processing; resolution is usually defined by [probable resolution in A of final structure] – 3-5A
important: frequency should be entered in 1/pixel units, to convert desired A resolution into
1/pixel frequency:
[desired resolution in A]/[A/pixel for your images]
We generally use Butterworth filter, for two frequency values enter [1/pixel frequency] ± 0.05

input files: vol/vol001.spi
output files: refprj/angles.spi, refprj/select.spi, refprj/prj*****.spi, refprj/d4prj*****.spi
description: creates projections of the reference volume vol001.spi according to equally spaced
in 3D space angles
important: before running this code create directory vol and copy there your reference volume,
name it vol001.spi. Make sure that dimensions of the reference volume are equal to dimensions
of particles images, otherwise use volpdsh.spi code

input files: micros/particles2.spi, refprj/prj*****.spi, ser02f/ser*****.spi
output files: apmq001.spi
description: compares particles images with a series of reference images (projections generated
by refprj.spi and their mirrors). It finds in-plane rotation angle and X, Y shifts which align each
particle image with the most similar reference image
important: the choice of parameters in ap mq is extremely important, sometimes it takes few
iterations before you find the right ones.
General guidelines for choosing parameters: John??

input files: apmq001.spi
output files: sort/apmqgrpminus**.spi, sort/apmqgrpplus**.spi

description: analyses apmq001.spi file: identifies which particles belong to each of the plus and
minus groups and writes this info to apmqgrpminus**.spi and apmqgrpplus**.spi

endline.spi (or endlinecomb.spi)
input files: sort/apmqgrpminus**.spi, sort/apmqgrpplus**.spi
output files: sort/endgrpplus001.spi, sort/endgrpminus001.spi
description: counts number of particles in each of the plus and minus groups, average number
of particles per group, and standard deviation; writes this info to endgrpplus001.spi and
endgrpminus001.spi. These output files will be used by plotm.spi and plotp.spi to generate visual
representation of particles distribution

plotp.spi (or plotm.spi, or plotcomb.spi)
input files: sort/endgrpplus001.spi
output files: sort/cndisplus.spi, sort/rcndisplus.spi
description: generates two images (one is inverse of the other) which show particles distribution
within plus groups

shrm.spi (calls p_rotate.spi procedure)
input files: apmq001.spi, ser02f/ser*****.spi
output files: ser02fa/ser*****.spi
description: rotates, shifts and mirrors images according to parameters from apmq001.spi
important: calls procedure p_rotate.spi, no changes in its code are needed

input files: apmq001.spi, ser02f/ser*****.spi
output files: ser02fmr/ser*****.spi
description: mirrors images which were sorted into minus groups by apmq001.spi and copies
the rest of images unchanged. Thus, in this data set plus and minus groups can be combined
since all the members of minus groups were mirrored.

angles.spi (calls p_angles.spi procedure)
input files: apmq001.spi, refprj/angles.spi
output files: vol/angles.spi
description: apmq.spi code identified which reference projection each particle image most
closely resembles. This code assigns 3 Euler angles to each particle image equal to those of the
reference projection it resembles.

input files: apmq001.spi, ser02fa/ser*****.spi, vol/angles.spi
output files: vol/vol_bpcg.spi
description: makes 3D reconstruction by back projection of particles images according to their 3
Euler angles found by angles.spi code

rmreffp.spi (or rmreffm.spi)
input files: sort/apmqgrpplus**.spi, ser02fa/ser*****.spi, ser02f/ser*****.spi
output files: sort/avg/avggrpplus**.spi, sort/avg/vargrpplus**.spi, sort/p/*
description: runs 10 rounds of reference-free alignment on particles from ser02f separately for
each plus (or minus) group, if no particles were assigned to particular projection will skip it

input files: sort/apmqgrpminus**.spi, sort/apmqgrpplus**.spi, ser02fa/ser*****.spi,
output files: sort/avg/avggrpcomb**.spi, sort/avg/vargrpcomb**.spi, sort/comb/*
description: runs 10 rounds of reference-free alignment on particles from ser02fmr separately
for each combined projection group. If no particles were assigned to particular projection will skip

input files: sort/p/*, sort/m/*, sort/avg/avggrpplus**.spi
output files: sort/pm/*, sort/avg/apsr***
description: runs 1 round of ap sr on 10 ap sr averages from rmreffp.spi and rmreffm.spi

input files: sort/avg/avggrpcomb**.spi, sort/apmqgrpcomb**.spi, sort/comb/*
output files: sort/comb/*
description: runs 1 round of ap sr on 10 ap sr averages from rmrefcomb.spi

input files: sort/comb/aapsr***.spi, refprj/angles.spi
output files: vol/vol_bpcgcomb.spi
description: creates 3D volume by back projecting ap sr averages
important: the code assumes that all projections are present in sort/comb/aapsr*, if not you need
to do some modification

For TILTED pairs (Random-Conical Reconstruction)

This method of data processing is based on the fact that single macromolecules often assume
preferred orientations on the specimen grid. Any subset of molecules showing identical views in
an untilted specimen form a rotation series with random azimuth. When the specimen grid is tilted
by a fixed angle, X, the above subset will appear in the micrograph as a conical projection series
with random azimuths and X as cone angle. On the microscope we record both the untilted and
tilted views/micrographs. The tilted micrograph is used to extract the projections for the
reconstruction. The untilted micrograph is used to (1) separate particles according to their views
(“hierarchical classification”) and (2) within each class (subset) determine the relative azimuths of
all particles (alignment). The tilted-particle images are used to obtain 3D reconstruction using
alignment information about their relative angles.

selecting particles from tilted pairs

description: when selecting particles you will match each particle from untilted micrograph with
itself on tilted micrograph, WEB will help facilitate the process
what to do:
1.        Open WEB window by typing >web4 spi
2.        Display the micrograph: COMMANDS -> TILTED PARTICLES -> [open untilted image
first, then tilted image] -> [Enter number for the output document, usually last 3 digits of tilted
micrograph name].
3.        Click on the mouse middle button, choose SHIFT IMAGES, look for similar patterns in the
two micrographs, when done with shifting click middle button again.
4.        Start selecting particles. Select the untilted particle (left) first, then select the
corresponding tilted particle (right).
5.        After picking 6-15 particle pairs, fit the angles: click on the mouse middle button, chose
FIT ANGLES, enter the appropriate particle for origin, then click DETERMINE THETA, FIT
ANGLES and SAVE ANGLES. Now when you choose left particle WEB will automatically
determine right particle. If precision gets off, repeat this step.
6.        Output files are: dct***.spi – tilted particles; dcu***.spi – untilted particles, dcb***.spi -

creating noise window
purpose: create image which contains only noise, it will be used to normalize all particles (see
what to do: go to WEB: >web4 spi &
1.      Display one of the un-decimated untilted micrographs (micros/****.spi)
2.      Go to EDIT -> Window Last File -> Fixed Size. Enter the window size: it should be
approximately 50% larger than the size of your particles (check different values until you find the
best one) and the name of noise-file: noise.spi
3.      The noise window should be selected by clicking on the characteristic portion of the
micrograph without any particles.
4.      To create d4noise.spi simply decimate noise.spi by 4 times

newwcut_t.spi (newwcut_u.spi)
input files: micros/microsdoc.spi, micros/noise.spi, micros/d4noise.spi, micros/dct***.spi
output files: ser/sert*****.spi, dser/dser*****.spi. micros/particles_t.spi
description: cuts particles from tilted micrographs according to coordinated recorded in dct***
important: you need to create noise files: noise.spi and d4noise.spi before running this code

identifying bad particles
purpose: eliminate obviously bad particles which were accidentally selected, thus clean the data
set for future data processing
what to do:
1.        In WEB go to COMMANDS -> Categorize -> Sequential Montage and display untilted
(tilted) particles from dser directory (usually 500 at a time), choose 1 category, name results file
../micros/handsort_u.spi (../micros/handsort_t.spi)
2.        Click on the bad particles, it does not matter in which order you click on them. After you
are done with the subset, clear the screen and display next portion (see step 1).
3.        Remember you need to do this separately for untilted and tilted particles and save results
correspondingly to handsort_u.spi and handsort_t.spi

input files: ser/*, dser/*
output files: reject/*
description: eliminates bad particles from ser and dser directories, even if only 1 particle in the
pair was identified as bad, both will be eliminated. Moves bad particles into directory reject

input files: ser/*, dser/*
output files: ser02/*, dser02/*, micros/handsort03.spi
description: renames consecutively particles in ser and dser directories, writes results to ser02
and dser02

input files: ser02/sert*****.spi, ser02/seru*****.spi
output files: fser02/*
description: filters tilted and untilted particles to specified resolution

input files: fser02/fseru*****.spi
output files: hier/avg***.spi, hier/temp***.spi
description: finds best alignment for untilted particles with respect to each other by cross
correlation, e.g. does one round of ap sr on them

input files: fser02/fseru*****.spi, hier/temp***.spi
output files: faser02/faseru*****.spi
description: rotates and shifts untilted particles according to parameters from apsr.spi

input files: faser02/faseru*****.spi
output files: hier/map001.spi (most important), hier/*
description: runs hierarchical clastering on self-aligned particles from faser02/faseru*****.spi
important: for better description of hierarchical clastering see either Spider website of Frank‟s

hierarmerge.spi (optional)

input files: hier/grp***.spi
output files: hier/group***.spi
description: use this code if you need to merge some groups from hierarchy.spi

input files: micros/particles_u.spi, micros/handsort03.spi, hier/grp***.spi
output files: hier/micnum.spi
description: determines to which micrograph particles from group you specified (hier/grp***.spi)

input files: hier/micnum.spi, hier/grp***.spi, micros/dcb***.spi, fser02/fsert*****.spi,
output files: hier/angles***.spi, hier/grp**/prjt*****.spi
description: creates file angles***.spi which lists angles for all tilted particles belonging to a
specified group (hier/grp***.spi)

input files: hier/grp01/prjt*****.spi, hier/angles***.spi
output files: hier/gr1bpcg.spi
description: creates 3D reconstruction of tilted particles according to angles found in

pjcc_gr1.spi (pjcc_gr2.spi)
input files: hier/grp01/*, hier/gr1bpcg.spi
output files: hier/gr1refine/*
description: runs specified number of rounds of refinement on tilted particles from hier/grp01/

For DEFOCUS pairs

Due to the basic principles of electron microscopy, close to focus micrographs have low signal to
noise ratio but contain important information that is missing from far from focus micrographs.
Thus, we developed a technique of defocus pairs. We take 2 pictures of the same spot, first close
to focus picture (defocus approximately -400-800 A), then far from focus picture (defocus
approximately -1900 A). We select particles from far from focus micrographs, but then cut
particles from close to focus micrographs. The following codes find parameters to match two
micrographs in a pair (by cross correlation), then adjust particles coordinates, which were for far
from focus micro, so that they will be particles coordinates for close to focus micro. Last code
eliminates particles which happened to be outside of range of low defocus micrograph.

input files: micros/microsdoc.spi, micros/d7i****.spi
output files: micros/diw****a.spi, micros/diw****b.spi, micros/alipair****.spi,
micros/alipairsuma.spi, micros/alipairsumb.spi, micros/alipairsumc.spi, micros/alipairsum_best.spi
description: correlates portions of 7 times decimated micros: finds inplane rotation and x,y shift
important: if this code crashes for unknown reasons and it looks like it is memory related try
running lowdefcorr_wstops.spi instead

input files: micros/microsdoc.spi, micros/alipairsum_best.spi, micros/handpick****.spi
output files: micros/hand2pick****.spi
description: applies inplane rotation and x,y shift to the coordinates of particles selected from
micrographs with high defocus (handpick****), will save results (coordinates of particles on close
to focus micros) to hand2pick**** - this file is for micros with the same decimation as
handpick****. Moves all handpick**** files to directory handpick_org (just in case something went
wrong and you‟ll need to use them)

input files: micros/microsdoc.spi, micros/hand2pick****.spi, micros/d4i****.spi or
output files: micros/handpick.spi
description: eliminates particles, which coordinates are outside of the micrograph range, will
save results to handpick****

input files: dfpmicros/handpick****.spi, dfpmicros/microsdoc.spi
output files: ser_dfp/*, dser_dfp/*
description: cuts particles from close to focus micrographs

For CTF determination

Image formation in the electron microscope is a complex process. The resulting image is not
necessarily a faithful representation of the object‟s projection, and hence the same can be said
for the relationship between the 3D reconstruction computed from such images and 3D object it is
supposed to represent. Contrast Transfer Function (CTF) estimation and application of CTF
correction are important for recovery of original information. Please read Frank‟s book and
collection of papers on CTF to understand image formation and Contrast Transfer Theory.

power.spi (calls power_p1.spi, which calls
input files: micros/microsdoc.spi, micros/i****.spi
output files: power/pw_avg****.spi, power/pw_ro****.spi, power/roo****.spi
description: calculates 2D, 1D and 1D displayable CTF functions for your micrographs, we
usually look at roo****.spi

input files: micros/microsdoc.spi, power/roo****.spi
output files: power/minima****.spi
description: finds minima in 1D displayable CTF function roo****.spi
important: sometimes it is unable to identify first minima because it is not pronounce enough,
thus ctfnonoise.spi won‟t be able to eliminate noise completely, you can either work with the
output file as is (you won‟t be able to determine Gaussian envelop half-width, but otherwise it‟s
fine) or you can handcorrect it.

input files: micros/microsdoc.spi, power/minima****.spi, power/pw_ro****.spi
output files: power/nonoise****.spi, power/noiserem****.spi
description: removes noise from pw_ro**** and creates CTF functions analogous to the ones
from power.spi but with the noise removed

input files: micros/microsdoc.spi
output files: power/defgrps.spi
description: creates template file which you will fill with actual values for defocus and Gaussian
envelope half-width for each micrograph

determining Defocus value and Gaussian Envelop ½ width
purpose: to determine defocus (underfocus) value for each micrograph and Gaussian envelop

1.       Go to WEB and open the power spectrum by using COMMANDS -> CTF FROM DOC
FILE -> power/noiserem****.spi
2.       Enter the pixel size in A per pixel (2.055 for ice)
3.       Enter 2.0 for the „Spherical aberration constant‟.
4.       Enter 120 for „Electron energy‟.
5.       Enter 0.09 for „Amplitude ratio‟.
6.       Choose Gaussian envelop half-width value it should be between 0.05 and 0.18, usually
0.09 or 0.1 work best, if you will need to try few numbers before you find the correct one
7.       A graphical representation of the power spectrum will display on the screen, the red line
is the reference defocus curve, and the green line is the experimental defocus curve. The
„defocus‟ and the „plot cutoff‟ are adjusted to obtain the best defocus values. Start fitting of the
experimental CTF to the calculated trace by using the course defocus value adjustment. Align the
minima of the experimental CTF (green) with the points of inflection of the calculated (red)
8.       The defocus and Gaussian envelop values should be written to defgrps.spi file which you
created in the previous step.

sorting micrographs into Defocus Groups
purpose: to group all micrographs into groups with similar defocus, we strive to have at least
500-1000 particles per defocus group and defocus spread < 500-800 A
what to do:
1.       Open power/defgrps.spi in Excel
2.       Sort by column with defocus values and group micrographs into defocus groups making
sure the defocus spread is not too large and at the same time there are enough particles per
group. Record defocus group number to second column.
3.       Sort particles back into their original order. Save file as defgrps.txt and copy it into power

input files: power/defgrps.txt, power/defgrps.spi
output files: power/defgrps_sorted.spi
description: takes text file defgrps.txt (created in Excel) and converts it into readable Spider

input files: micros/microsdoc.spi, power/defgrps_sorted.spi
output files: power/defgrpsstat.spi
description: creates summary file for defocus groups, which contain number of particles in the
group, average defocus, Gaussian envelop half-width, defocus spread (usually set to 0, but no
more than 100)

input files: refine/order_select001.spi (can be substituted by power/defgrpsstat.spi)
output files: results/ctf/*
description: calculates 1D displayable CTF function for the whole dataset


Common lines is a method for determining the relative 3D orientation of >= 3 projections. For any
two projections of the same object, central sections of the 3D Fourier transforms for those
projections will intersect each other along a common line through the origin (Frank, 2006). Once
the common line between two central sections (and thus, two projections) is found, their
movement relative to each other becomes constrained by the fixed common line. When the
common line between the second projection and a third projection is established, all positions will
become fixed and relative angles between all three projections can be determined.

input files: apsrselect01.spi, classavgs/classavg**
output files: rd**/, rd**/angles/rangles001.spi, rd**/angles/rangles002.spi, rd**/vol001,
description: determines relative Eulerian angles for a set of projections and backprojects a 3D

input files: rd**/angles/rangles002.spi, rd**/angles/angles**, ptcls/grp**/prjt*****
output files: rd**/angles/anglesnew**
description: assumes that (1) tilted particles corresponding to 3 or more different untilted views
exist, (2) the Eulerian angles for each tilted particle have been determined, (3) a class average
exists for each untilted view, (4) the relative 3D orientations between those class averages have
been determined by common lines. This procedure will adjust the Eulerian angles for all tilt
particles, from as many different untilted views as are present, such that all tilt particles‟ new
angles are relative to a common axis. Each grp** directory contains the tilt particles (prjt*****) for
a given class average, and the corresponding angles** file contains angles for each tilt particle.


input files: rd**/angles/angles**, ptcls/grp**/prjt*****
output files: rd**/angles/anglesnewcomb.spi, rd**/ptcls/prjt*****,
description: creates a combined angles file and a directory containing all tilt particles


input files: rd**/angles/anglesnewcomb.spi, rd**/ptcls/prjt*****
output files: rd**/vol002.spi
description: backprojects a volume from all all tilt particles based on combined angles


description: combines op.spi, rctcomb.spi, ptclsort.spi, and bprpcomb.spi ; performs as many
independent rounds of common lines as specified

For Reference-Free Alignment and Classification


input files: counter.spi, ser*****
output files: rd**/**temp***, rd**/**avg***, rd**/ser*****
description: performs reference-free alignment for a set of particles and aligns particles based
on alignment parameters as determined by Spider command AP SR


input files: rd**/ser*****
output files: rd**/coran_***, rd**/kmeans, rd**/kmcluster**
description: performs Multivariate Statistical Analysis and either (1) K-means or (2) Hierarchical
Ascendant Classification on a set of particles; for K-means, the number of class averages must
be set by the user; for HAC, the number of classes is determined by the selected threshold value


input files: rd**/coran_IMC, rd**/ser*****
output files: rd**/kmclusteravg**
description: for either method of classification, creates class average images for each class


description: combines apsr4class.spi, classify.spi, and classavg.spi

Other Spider Codes

usage: extension.script old_ext new_ext
description: will change extensions of all files in the current directory with old_ext to new_ext

input files: files*****
output files: files*****
description: orders files inside a directory, so that files will be consecutively numbered
important: old numbering will be lost

input files: vol01
output files: vol_out
description: adjusts A/pixel and size for given volume to specified values

Description of the Refinement code


Shared By: