PARADISEC WORKFLOW MODEL by vwt15444

VIEWS: 26 PAGES: 11

									                             PARADISEC AUDIO WORKFLOW MODEL
                            As amended during workshop, 7 September 2005

                                   DRAFT for DISCUSSION
                                    Friday, September 9, 2005
1.   Pre-ingestion (audio tapes) ................................................................................................. 2
2.   Wavelab Ingestion Processes.............................................................................................. 5
3.   Wavelab processes for generation of derivatives (mp3 and CD-audio quality) (Audio) ... 7
4.   Rimage processes for burning CDs (admin)....................................................................... 8
5.   Quadriga Processes for BWF generation............................................................................ 8
6.   Archive process ................................................................................................................ 10
7.   Restore from archive process ........................................................................................... 10
8.   Create a static repository in XML .................................................................................... 10
9.   Check PARADISEC repository reports ........................................................................... 11

NOTE: Workflows in progress: digital origin ingestion; dynamic harvesting to the web.
NEED workflows for images, digital audio, video, film etc.

This document reflects practices in the audio ingestion unit of PARADISEC, the Pacific and
Regional Archive for Digital Sources in Endangered Cultures. For further information on
PARADISEC please visit our website http://paradisec.org.au.
Contact:

This document last updated February 22, 2006.

For questions or comments on this document please contact admin@paradisec.org.au, or write
to us at:

PARADISEC
Transient Building F12
Fisher Road
University of Sydney NSW 2006
Tel: +61 2 9036 9557
Fax: +61 2 9351 7572

Role terms used in this document
Admin: project administration officer (2005: Amanda Harris and Miriam Corris)
Audio: audio preservation officer (2005: Frank Davey)
PM: project manager (2005: Nicholas Thieberger, University of Melbourne)




                                                                                                                                      1
1. Pre-ingestion (audio tapes)
   1.1. assess collection (Admin and PM)
      1.1.1. assess whether material is suitable for accession (PM in consultation with
           steering committee)
          1.1.1.1.    rights
              1.1.1.1.1.         does the depositor assert ownership of the material?
              1.1.1.1.2.         are the rights in the material clearly specified?
          1.1.1.2.    content criteria (assess against policy) –
              1.1.1.2.1.         is the material unique?
              1.1.1.2.2.         is the country and language of the material known, and in the
                         core area (non-Australian)?
              1.1.1.2.3.         is there an alternative place for deposit of the recording?
              1.1.1.2.4.         is the content of the recording in a high-risk language?
              1.1.1.2.5.         is the format or recording medium at high risk?
          1.1.1.3.    institutional affiliation – give preference to consortium members
          1.1.1.4.    ease of integration into our systems
              1.1.1.4.1.         is data in a format we can handle (i.e. cassette, 1/4 inch reels)?
                  1.1.1.4.1.1.            if not, direct prospective depositor to suitable alternative
                               facility
              1.1.1.4.2.         consistency and adequacy of any metadata
      1.1.2. if collection assessed as suitable, create a record in the SQL database of the
           form [collection ID]-ASS
      1.1.3. assess tape condition and prepare material for transportation accordingly (see
           below)
   1.2. Transportation
      1.2.1. if the recordings are in bad condition (e.g. mould, rare formats, if tape visibly
           damaged) send to Sydney (contact Admin)
          1.2.1.1.    transport any mouldy items in separate ziplock bags and do not pack
                 with non-mouldy items
      1.2.2. if in reasonable condition transport to nearest ingestion point



                                                                                                      2
   1.2.3. prefer personal transport; transport in hand luggage if flying
   1.2.4. if necessary use registered courier
   1.2.5. packing – make sure end fastened on reel tape, pack tightly, wrap in bubble
         wrap, ensure against unnecessary shocks during transport
   1.2.6. transport with inventory of contents
   1.2.7. receiving ingestion point – check inventory against contents and contact
         depositor if any discrepancy, tick „tape received‟ in SQL database
   1.2.8. assess whether mould-affected and if so implement appropriate procedure (put
         in ziplock bags and store in „dirty room‟)
1.3. metadata
   1.3.1. where possible get depositors to enter own metadata in SQL database
       1.3.1.1.     assign login (admin)
       1.3.1.2.     associate assessed collection record with user, and change item ID for
                first record (i.e. change –ASS part of PI to the ID of the first item e.g. LB1-
                ASS > LB1-001)
       1.3.1.3.     email depositor with instructions
   1.3.2. where depositor cannot enter own metadata, create records (admin)
   1.3.3. assign Persistent Identifier (PI) – if depositor already has a consistent naming
         system, use it, or else use numerical symbols, e.g. 001
   1.3.4. label all tapes and tape cases to show PI, ensuring that no written information
         is covered up.
   1.3.5. sort all tapes in numerical order in filing cabinet (if to be digitised shortly) or in
         safe (if filing cabinet is full), or in dirty room if mould-affected
   1.3.6. tick „metadata entered‟ and „tape labelled‟ in SQL database
   1.3.7. when tapes returned, note date in „tracking‟ field in SQL database
1.4. prepare tapes for ingestion (Audio)
   1.4.1. assess tape condition – treat mouldy items according to procedure (wear
         protective gear)
       1.4.1.1.     dehumidify in vacuum oven (Audio) if necessary, with separate batches
                for mould-affected items



                                                                                              3
                1.4.1.1.1.       decide duration of treatment and temperature following advice
                         from NFSA
                         <http://www.screensound.gov.au/screensound/screenso.nsf>
                1.4.1.1.2.       if mouldy remove any visible growths first with cotton bud or
                         pad and solution of isopropyl and cetramide
                1.4.1.1.3.       if sticky shed no special prior treatment needed
         1.4.2. clean and prepare for ingestion (Audio)
            1.4.2.1.     reel-to-reel
                1.4.2.1.1.       repair or replace lead tape and damaged splices if necessary
                1.4.2.1.2.       clean using slow spool (Audio) using specialist equipment
                         (cotton pads and isopropyl/cetramide solution); also clean flange
                         (spool) separately
                1.4.2.1.3.       second slow spool to repack tape
            1.4.2.2.     cassette and DAT
                1.4.2.2.1.       replace splices if necessary (cassette)
                1.4.2.2.2.       fast forward and rewind to minimise print-through (cassette) and
                         to repack tape (cassette and DAT)
                1.4.2.2.3.       if mouldy, bag up separately, to avoid cross-contamination
                         shelve in dirty room and minimise time in ingestion room, replay on
                         „dirty‟ cassette machine.


format                           device                           comment
Cassette                         TASCAM 122 Mk II                 tested OK
Mouldy cassette                  Yamaha KX-W321                   Tested OK
Reel-to-reel quarter-track       Studer A807 only                 Under repair September
                                                                  2005
Reel-to-reel 15ips               Studer A807 or A810              tested OK
Reel-to-reel 7 1/2 ips           Studer A807 or A810              tested OK
Reel-to-reel 3 3/4 ips           Studer A807 or A810              tested OK
Reel-to-reel 1 7/8 ips           Currently attempting to          As at September 2005


                                                                                                 4
                                 locate suitable machine
Minidisk                         Processed in Melbourne –        Tested ok
                                 Minidisk Transfer Editor
*.wav files                      via network from                tested OK
                                 Melbourne, ANU or APAC
DAT                              Sony PCM 7040                   tested OK
CD                               CD inspector on Audiocube       Minor bug (September
                                 3                               2005) – Syncrotech
                                                                 investigating; tested OK
                                                                 October 2005




2. Wavelab Ingestion Processes
     2.1. pre-monitor audio levels and set ingest at appropriate level
     2.2. set up Wavelab to save the digitised audio to the file „D:WAV96 ingest:[PI].wav‟ on
         the D drive of Audiocube 1 or 3. Relevant sub-parts of an item are indicated „-A‟, „-B‟
         and so on, and are inserted after the PI and before the extension (this is necessary for
         audio files that will be over 2GB - about 55 minutes at 24-bit/96khz).
     2.3. after ingestion finished, trim silence at beginning and end of file, leaving margin of 5
         seconds‟ silence
     2.4. create programme markers to mark the beginning and end of the file, as well as where
         the tape recorder has been switched on and off (“transition points”) (duration between
         transition points must be greater than 4 seconds to allow audio CD tracks to be
         created)
     2.5. analyse the file in Wavelab (control-Y) (takes about 2 minutes for normal length file)
         - this generates temporary error markers wherever recording faults are apparent
     2.6. check the errors: if they are analogue errors make a note of them in SQL web
         catalogue; if they are digital errors, re-ingest the recording.



                                                                                                     5
2.7. if necessary, normalise (takes about 10 minutes). This should not normally be
    necessary, as Audio performs a level check before beginning to record.
2.8. save the file (takes about 2 minutes or so). This also creates a *.MRK file that stores
    the marker information, with points marked in samplecounts.
2.9. enter info into SQL database
   2.9.1. Audio Notes (use „<p>‟ for linebreaks that need to be kept)
   2.9.2. tick digitised „yes‟ (which auto-enters date digitised)
   2.9.3. operator info
   2.9.4. tape speed
   2.9.5. if recorded tape ID mentions date created, tape ID, recordist, or any other info
         not already in database fill in relevant fields (check with admin as necessary, if
         unsure, leave blank)
   2.9.6. Matching filenames harvested nightly from „WAV 96khz for BWF‟ and bitrate
         and samplerate info entered from matching files in „XMLExport‟
2.10.      check that mandatory metadata fields (PI, title, date created, country,
    collector) are in SQL catalogue, if not, add them
2.11.      when mandatory fields present, copy the audio file *.wav to
    „Azoulay:Digitization:WAV 96khz for BWF‟
2.12.      copy the marker file *.MRK to „Azoulay:Digitization:mrk txt files‟
2.13.      move the folder containing the day‟s work files as a backup to „Azoulay:Audio
    staging area‟




                                                                                               6
3. Wavelab processes for generation of derivatives (mp3 and CD-audio quality) (Audio)
   3.1. run Wavelab batch processor on files in „D:Sent to Azoulay - convert to 44.1‟ to
       downsample the 24bit/96khz „*.wav‟ file to 16bit/44.1khz (resultant files named „*
       CD 44-1.wav‟ created in directory „D:WAV44_1 CD‟)
      3.1.1. Wavelab batch processor also normalises to –1db and dithers to 16-bit
   3.2. Wavelab batch processor to create 128kbps MP3 files
      3.2.1. rename mp3 files to remove „_CD‟ from filename
   3.3. if creating audio CDs in Wavelab, run Wavelab batch processor „autosplit‟ function
       using the preset „generic marker to CD marker‟ to create a „* CD 44-1.MRK‟ file with
       the samplecounts changed to match the new sample rate of 44.1 (also stored in the
       „WAV44-1 CD‟ folder)
   3.4. if creating audio CDs in Rimage run Wavelab batch processor „autosplit‟ function to
       create separate audio files for each track, create separate folder for each master CD
       file
   3.5. Set up within batch process to limit size of folders to 650MB, limiting CDs to a single
       PI or a single master file
   3.6. copy the wav and *.MRK files to directory „Azoulay:Digitization:WAV 44-1 for CD‟
   3.7. copy the mp3 files to Azoulay:Archive:Sealed




                                                                                               7
4. Rimage processes for burning CDs (admin)
   4.1. copy the files in „Azoulay:Digitization:WAV 44-1 for CD‟ downloaded to burning
       machine HD „D:CDs‟
   4.2. for data CDs
      4.2.1. start „new project‟ in QuickDisc, select „data CD‟, drag files into programme
      4.2.2. create label in CD Designer, and select it for that project in QuickDisc
      4.2.3. load CDs into lefthand bin
      4.2.4. submit to burn (typically, 2 copies, but programme will handle up to 100
            copies)
      4.2.5. when finished put in cases
   4.3. for audio CDs
      4.3.1. start „new project‟ in QuickDisc, select „audio CD‟, drag files into programme
      4.3.2. create label in CD Designer, and select it for that project in QuickDisc
      4.3.3. load CDs into lefthand bin
      4.3.4. submit to burn (typically, 2 copies, but programme will handle up to 100
            copies)
      4.3.5. when finished put in cases
   4.4. tick „CD burned‟ in SQL; if using gold CD stock, enter ID numbers in „CD ID‟ field
   4.5. return CDs to researcher with „processed and archived by PARADISEC‟ slip and
       letter listing inventory of CDs returned, with instructions on how to access files online
   4.6. note return of CDs in „tracking‟ field in SQL database
   4.7. move files from 'Azoulay:Digitization:WAV 44-1 for CD' to
       'Azoulay:Digitization2:burned CDs' and delete files from burning machine

5. Quadriga Processes for BWF generation
   5.1. start AFI on Audiocube 2 (BWF file *.tmp.wav combining the *.imp.xml and *.wav
       files is processed to TempBWF folder)
   5.2. open *.tmp.wav file in BWF editor on Audiocube 2
   5.3. press „b‟ to open the metadata header, select „quality extension‟ tab


                                                                                              8
5.4. fill in fields „inspector‟, filestate=Y, quality (0>5) and paste audio notes from SQL
    database into „operator comment‟ field
5.5. run BWF editor on files in TempBWF folder to check data is OK (triggers Quadriga
    Batch processor on Audiocube 2 to begin automatic process of creating *.wav file in
    PresealedBWF folder and *.exp.xml file in XMLExport folder)




                                                                                             9
6. Archive process
   6.1. automatic Quadriga Batch process then begins, to seal the BWF and store it in
       „Azoulay:Archive:SealedBWF‟ folder
   6.2. periodically the APAC store will look at the Azoulay:Archive:SealedBWF folder and
       copy any new files to the APAC store
   6.3. when the Azoulay:Archive partition gets to 60% full it should trigger an automatic
       archive process to the MSS at Sydney Uni, removing the oldest files in order, to take
       the capacity down to say 20%
   6.4. the archive process will store the batch of files on 200GB digital tape under a single
       archive tag; the tag and the names of files archived are emailed to everyone
   6.5. Import reports into catalogue by cut-and-pasting into the „upload Azoulay archive‟
       section of the Import/Export tab in catalogue
   6.6. as part of the archive process, the archive is proven by duplicating the tape and then
       the originals are deleted from Azoulay:Archive
   6.7. the duplicate tape is returned to the PARADISEC lab to be stored in the safe
   6.8. when an archive tape has been produced a report is sent out automatically. The
       contents of this message should be put into the 'Upload Azoulay archive report' box
       on the import/export screen and uploaded.

7. Restore from archive process
   7.1. At present restores can only be done by Matthew Geier. It is advisable to do periodic
       restore from archive every six months or so.

8. Create a static repository in XML
   8.1. Generate file using export routine from SQL
   8.2. Open the webserver (Caligula at ANU) and rename existing PDSCsr.xml file as
       PDSCsrYYYYMMDD.xml using the 'date modified' of the existing file on caligula
   8.3. Copy the newly generated file to a text file and name it PDSCsr.xml
   8.4. Move this file into caligula


                                                                                                 10
   8.5. Go to the OLAC Archive registration site "http://www.language-
       archives.org/register/register.php4" and enter paradisec.org.au/PDSCsr.xml
   8.6. Click on validate and wait for the process to finish
   8.7. If errors occur, check on what they are, edit the PDSCsr.xml document to correct
       them and go back to step 8.4, overwriting the current PDSCsr.xml file.

9. Check PARADISEC repository reports
   9.1. Check PARADISEC repository audit report emailed weekly to all project members
       from stuart.hungerford@anu.edu.au (Audio)
   9.2. Identify any anomalies and rectify, reprocessing relevant files as necessary.




                                                                                           11

								
To top