Docstoc

1.1 IceBridge Provider Workflow

Document Sample
1.1 IceBridge Provider Workflow Powered By Docstoc
					                                                  IceBridge SIPS Operational Concept
                                                                        IceBridge Provider - NSIDC DAAC Interface

  20110316_142120_ATM4BT2.spatial
                                                   IceBridge Provider Workflow
   76.534, -68.705
   76.545, -68.706
   76.546, -68.707
   ...                                                          Metadata
                                                                                           20110316_142120.ATM4BT4.qi.met
                                                                Generator


        20110316_142120_ATM4BT2.pre_met

  Version            = "1"
  BeginDate          = "2011-03-16"                          20110316_142120.ATM4BT4.qi              PDR Generator
  EndDate             = "2011-03-16"
  BeginTime          = "14:21:04.697Z"


                                                                                                                                   Data Flow
  EndTime            = "14:27:32.763Z"
  CampaignName       = "2011_GR_NASA"
  AircraftType       = "DC-8"
                                                                                           20110316_142120.ATM4BT4.qi.pdr
  AircraftID         = "N817NA"
  Instrument          = "ATM"


                       Valids.cfg
                                                                                                                                   Via FTP or Media Delivery
                                                                                                     Product Delivery
   CampaignName      = "2009_GR_NASA"
   CampaignName      = "2010_GR_NASA"
   CampaignName      = "2011_GR_NASA"
   AircraftType      = "DC-8"
   AircraftType      = "P-3B"
   AircraftType      = "B-200"
   AircraftID        = "N817NA"
   AircraftID        = "N80596"
   AircraftID        = "N3148M"                                                                           OR
   Instrument        = "ATM"
   Instrument        = "DMS"
   Instrument        = "LVIS"



                          ATM4BT2.cfg                                                                                     FTP
  DataFileCount = 1
  DataFileExtension1 = ".qi"
  QA = "N"                                                                                    Provider provided (per data file)
  MCFFileLocation = "/data/ATM/ILATM1B#001.mcf"
  ValidsLocation = "data/ATM/Valids.cfg"                                                      NSIDC DAAC provided (per data set)

                                                                                              SIPS tool




                                                                                                                                                          IceBridge NSIDC DAAC Workflow



                                                                                                                                                       20110316_142120.ATM4BT4.qi

                                                                                                                                                       20110316_142120.ATM4BT4.qi.met

                                                                                                                                                       20110316_142120.ATM4BT4.qi.pdr
                                                                                                                                                  OR
                                                                                                                                                                Provider 1



                                                                                                                                                                                                 ECS
                                                                                                                                            FTP


                                                                                          Ingest Status                                                         Provider 2

                                                                                          Via Email or File                                                                             Ingest Status via PAN &
                                                                                                                                                                                           PDRD Messages




                                                                                                                                                                                                  OR

                                                                                                                                                                                        Email



                                                                                                                                                           ...  Provider 3                                        File




                                                  Figure 1-1 IceBridge Provider – NSIDC DAAC Interface


1.0                    Operations Scenario


A nominal data flow between an IceBridge Data Provider and the NSIDC DAAC (see Figure 1-
1) will consist of the provider sending (via FTP or media) their data files and associated metadata
and Product Delivery Record (PDR) files to the NSIDC DAAC. These associated files are
generated by the provider, using the Science Investigator-led Processing System Metadata
Generation (SIPSMetGen) tool. These files are to be ingested into the DAAC’s archive system
EOSDIS Core System (ECS). Following an attempted ingest, ECS reports status via two types of
message files (see Figure 1-2). If the PDR file has errors, a Product Delivery Record
Discrepancy (PDRD) is created and the data are not ingested into ECS. If the PDR is valid, but
there are problems with the data file or metadata file, a long Product Acceptance Notification
(PAN) is created with basic failure information in it, and again ECS ingest will fail. If there are


                                                                                                                                   1
                  IceBridge SIPS Operational Concept
no errors in any of the delivered files (data, metadata, or PDR), ingest occurs successfully and
the PAN is used to confirm the successful status to the provider.

Data can be staged for online delivery in two ways: using NSIDC as a staging area, or through a
polling server on the provider’s system. If NSIDC is used as a staging area for the providers and
ECS ingests the files from this local staging area, the ECS system with operator control, can
automatically clear the data from the staging area after confirming successful ingest through the
existence of successful PANs. In this case, the operations staff also reviews PDRDs and Long
PANs which indicate errors. NSIDC will evaluate failures and work with the Providers when
necessary to fix the issues and then attempt to ingest the data again.

 If a provider chooses to set up a polling server (an ftp server where the relevant files are staged
for download) for ECS to poll directly, then the notification files (PANs and PDRDs) can be
delivered directly to that polling server via ftp or email. In this scenario, the provider will be
responsible for removing the staged data that was successfully ingested (as noted through the
successful PANs), and will need to work with NSIDC to correct ingest failures, noted through
long PANs.

                                  IceBridge NSIDC DAAC Workflow



                               20110316_142120.ATM4BT4.qi

                               20110316_142120.ATM4BT4.qi.met

                               20110316_142120.ATM4BT4.qi.pdr
             OR
                                        Provider 1


                                                                                    ECS
  FTP


                                        Provider 2
                                                                           Ingest Status via PAN &
                                                                              PDRD Messages




                                                                                     OR

                                                                           Email



                                  ...   Provider 3                                                   File




                                                            2
IceBridge SIPS Operational Concept
       Figure 1-2 IceBridge NSIDC DAAC Workflow




                      3
                          IceBridge SIPS Operational Concept
20110316_142120_ATM4BT2.spatial
                                                IceBridge Provider Workflow
 76.534, -68.705
 76.545, -68.706
 76.546, -68.707
 ...                                                           Metadata
                                                                                        20110316_142120.ATM4BT4.qi.met
                                                               Generator


      20110316_142120_ATM4BT2.pre_met

Version      = "1"
BeginDate    = "2011-03-16"                                20110316_142120.ATM4BT4.qi             PDR Generator
EndDate       = "2011-03-16"
BeginTime    = "14:21:04.697Z"
EndTime       = "14:27:32.763Z"
CampaignName = "2011_GR_NASA"
AircraftType = "DC-8"
                                                                                        20110316_142120.ATM4BT4.qi.pdr
AircraftID    = "N817NA"
Instrument    = "ATM"


                   Valids.cfg
                                                                                                 Product Delivery
 CampaignName      = "2009_GR_NASA"
 CampaignName      = "2010_GR_NASA"
 CampaignName      = "2011_GR_NASA"
 AircraftType      = "DC-8"
 AircraftType      = "P-3B"
 AircraftType      = "B-200"
 AircraftID        = "N817NA"
 AircraftID        = "N80596"
 AircraftID        = "N3148M"                                                                          OR
 Instrument        = "ATM"
 Instrument        = "DMS"
 Instrument        = "LVIS"



                      ATM4BT2.cfg                                                                                    FTP
DataFileCount = 1
DataFileExtension1 = ".qi"
QA = "N"                                                                                   Provider provided (per data file)
MCFFileLocation = "/data/ATM/ILATM1B#001.mcf"
ValidsLocation = "data/ATM/Valids.cfg"                                                    NSIDC DAAC provided (per data set)

                                                                                           SIPS tool



                                    Figure 1-3 IceBridge Provider Workflow


1.1         IceBridge Provider Workflow


All IceBridge data providers will be provided a tool to facilitate the following:

            1. Generation of an ECS-compatible metadata file (the .met file)
            2. Generation of ECS instructions on what to ingest (the .pdr file)
            3. Optional delivery of all the files associated with a data granule to NSIDC. This option
               is only available when the data are to be transferred to a staging area at NSIDC via
               the internet (using FTP, or possibly SCP).



1.1.0 SIPSMetGen Tool Preparation



                                                           4
                IceBridge SIPS Operational Concept
The SIPSMetGen tool will require the following input files specifically for each science data file:

   1. Provider provided files
         a. Data file(s) (ex: 20110316_142120.ATM4BT4.qi) – this file represents a science
             product to be ingested at the DAAC.
         b. Spatial file (ex: 20110316_142120.ATM4BT4.spatial) – this is an ASCII,
             comma- delimited file of spatial coordinates (latitude, longitude) the provider will
             extract from their data file. We expect that the number of coordinates in the file
             will be handled by the SIPSMetGen tool as long as the data files are not over the
             recommended 1GB size. As the data file sizes approach the 2GB maximum, some
             hosts may need large amounts of memory to handle the spatial processing.
             Accuracy and precision of this data should not matter within reason as the spatial
             algorithm will create a polygon of appropriate precision.
         c. Metadata file (ex: 20110316_142120.ATM4BT4.pre_met) – this is an ASCII file
             containing 10 parameters of the form: name = “value”. Value must be surrounded
             by double quotes. The 10 parameters are:
                   i. Filename: this is the filename of the product to ingest. Providers will be
                      given guidelines on how to best construct filenames for use with the
                      SIPSMetGen tool. Specific format may vary.
                  ii. Version: this parameter represents the processing version for the current
                      provider product. This is a local version ID that can be used to specify
                      small changes in the production of the product etc…
                iii. BeginDate: this is the beginning date when data in each datafile was
                      acquired in the form “YYYY-MM-DD”
                 iv. EndDate: this is the ending date when data in each datafile was acquired in
                      the form “YYYY-MM-DD”
                  v. BeginTime: this is the beginning time when this data was acquired in the
                      form “hh:mm:ss.sssZ”. Time is in Zulu or UTC.
                 vi. EndTime: this is the ending time when data was acquired in the form
                      “hh:mm:ss.sssZ”. Time is in Zulu or UTC.
                vii. CampaignName: this is the official name of the particular flight campaign
                      when this data was acquired. See the “ValidKeywords.cfg” file for the
                      accepted names.
               viii. AircraftType: this is the official name of the particular aircraft used to
                      acquire this data. See the “ValidKeywords.cfg” file for the accepted
                      names.
                 ix. AircraftID: this is the official name of the particular aircraft ID found on
                      the aircraft used to acquire this data. See the “ValidKeywords.cfg” file for
                      the accepted names.



                                                5
           IceBridge SIPS Operational Concept
              x. Instrument: this is the official name of the particular instrument used to
                 acquire this data. See the “ValidKeywords.cfg” file for the accepted
                 names.



2. NSIDC DAAC provided files
      a. ValidKeywords.cfg – this file contains a list of all valid keywords used for the
         following metadata attributes:
               i. CampaignName
              ii. AircraftType
             iii. AircraftID
             iv. Instrument
      b. SIPSMetGen.cfg – The contents of this will likely be unique to each data set. This
         file contains pathnames to the data product files (data, browse, etc…) so the tool
         can find them. Output paths for the metadata and pdr files, as well as other
         configuration parameters required for operation of the tool such as minimum grid
         cell size, max grid cell count etc...

          A sample will be delivered and each provider can tailor and name this
          configuration file appropriately for each data set that requires different
          parameters.

       c. NSIDC_Paths.cfg - The contents of this file will be unique for each provider. This
          will contain paths that will go into the PDRs which NSIDC’s polling service will
          use to find the data on the polling location. (Needed for local polling and Provider
          polling server.) This file also contains the FTP address, login information, and
          path to the NSIDC DAAC staging area when files are uploaded by the
          SIPSMetGen tool.
       d. Metadata Configuration File (ex: ILATM1B#001.mcf) – this file is essentially a
          product-specific metadata template that the SIPSMetGen tool uses to create ECS
          compatible metadata. Each product will have a unique .mcf file.

       REQUIREMENT: The SIPSMetGen Tool requires that every file (science file(s),
       browse, product history (PH), Quality (QA), spatial file, and pre-met file) associated
       with the data file begin with the same naming convention. That name must be unique
       to each file in the data set and should include a beginning collection date and time for
       the data in each data file. File extensions can be changed and configured in the
       SIPSMetGen.cfg file.

       There is a sample of an existing data type below. The format of the name of this data
       set is acceptable.


                                            6
                 IceBridge SIPS Operational Concept
            The ideal naming convention would begin with a word uniquely identifying the data
            or instrument, an indication of data product level, date of collection, start time of the
            data file’s collection time period.

            Ideal Example: ATM4BT4_ATM1B_20110316_14120.qi




            Acceptable Existing Example:

            (Following the example given above for existing qfit data)

            20110316_142120.ATM4BT4.qi

            20110316_142120.ATM4BT4.spatial

            20110316_142120.ATM4BT4.pre-met

            The unique file name is then: 20110316_142120.ATM4BT4

            Subsequently if browse data, PH or QA files were created they must begin with this
            unique file name and contain an appropriate file extension (e.g. Browse:
            20110316_142120.ATM4BT4.brws).

1.1.1 SIPSMetGen Tool Processing


When all the provider files, configuration files and NSIDC-provided files are in place and
configured appropriately, the tool is ready to run. The SIPSMetGen tool will be a java-based
command line tool, developed on a Linux platform.

Run Examples include three scenarios of operation:

Scenario 1: process the files and push data to a staging area at NSIDC

# /path/SIPSMetGen.exe /data/2009AntarcticaData/ –c
/data/OurRadarData/configs/OurData1B.001.config –C 2009_AN_NASA -P

This command runs SIPSMetGen.exe, looks for data files in provider’s local directory
/data/2009AntarcticaData, uses information from the configuration file OurData1B.001.config for the
2009_AN_NASA campaign, and will attempt to push the data to NSIDC when complete.

Scenario 2: assuming files have already been processed, push them to a staging area at NSIDC

# SIPSMetGen.exe –c /data/OurRadarData/configs/OurData1b.001.config –C 2009_AN_NASA –PO


                                                    7
                  IceBridge SIPS Operational Concept
In this case, the SIPSMetGen tool would generate a list of unique identifiers based on the PDRs in the
PDROutputPath location, and then look in the data, browse and met file locations for matches and push
the respective files. The utility will push the PDR file last. This is also a scenario that might come up if
the provider were pushing data to a staging area at NSIDC.

Scenario 3: process the files, but do not attempt to push them to NSIDC

# SIPSMetGen.exe –c /data/OurRadarData/configs/OurData1b.001.config –C 2009_AN_NASA

In this case, the SIPSMetGen tool would not attempt to transfer the data. Instead, the provider could either
stage the files onto their own staging area for NSIDC to use as a polling server (an ftp server from which
NSIDC would download the files as they do in the standard SIPS model) or the provider could
subsequently copy the files onto hard drives (i.e. USB hard drives) formatted in ext3 to ship to NSIDC.

Specifics of flags etc… are subject to change but are given here as an example of how the tool might be
executed.




                                                      8

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:0
posted:4/13/2014
language:Latin
pages:8