ned_poster

Document Sample
ned_poster Powered By Docstoc
					                              NPOESS Enhanced Description Tool - “ned”
                 SDR  Sensor Data Record  TDR  Temperature Data Record  EDR  Environmental Data Record  IP  Intermediate Product  ARP  Application Related Product  GEO  Geolocation 

                                                           NPOESS  National Polar-orbiting Operational Environmental Satellite System  NPP  NPOESS Preparatory Project 




                                                                                                                                                                                                    Enhancements by “ned”
                          HDF5 for NPOESS                                                                                                                                                                                                                                                                                                                                                                                                           What is “ned”?
                                                                                                                                                                                                                                                                                                                                                                                                                              •      A “c” language demonstration exercise
          •    Hierarchical Data Format 5 (HDF5) is the format for                                 HDF5 NPOESS                                                                                                                                                        HDF5 NPOESS
                                                                                                                                                                                                                                                                                                                                                                                                                              •      An exploration of using the HDF5 API and other tools to
               delivery of processed products from the National Polar-                                                                                                                                                                                                      <<HDF5 Group>>       1                             <<HDF5 Group>>            1

                                                                                                                                                                                                                                                                                                                                                                                                                                     navigate the NPOESS product information model.
                                                                                                        <<HDF5 Group>>          1                                   <<HDF5 Group>>     1
                                                                                                                                                                                                                                                                                     / (root)                             1         All_Data
                                                                                                                 / (root)                                   1          All_Data

               orbiting Operational Environmental Satellite System                                 •attributes
                                                                                                                                                                                                                                                                       •attributes




               (NPOESS) and for the NPOESS Preparatory Project                                                   1
                                                                                                                                                                                                                                                                                     1
                                                                                                                                                                                                                                                                                                                                                                                                                                      – How hard is it to deal with the format challenges?
                                                                                                                                                                                                                                                                                                                                                                                                                                         The challenges marked with  (see left) are
                                                                                                                                                                                                                                                                                          1                                                     Bit-wise flag fields are                           *
                                                                                                                     1                                                                                                       *                                                                                                                   unpacked, split into
                                                                                                                                                                                                                                                                            <<HDF5 Group>>

               (NPP).                                                                                   <<HDF5 Group>>
                                                                                                                                                                                                                    <<HDF5 Group>>                                            Data Products
                                                                                                                                                                                                                                                                                                                                                 separate fields, then
                                                                                                                                                                                                                                                                                                                                                     compressed
                                                                                                                                                                                                                                                                                                                                                                                         <<HDF5 Group>>
                                                                                                                                                                                                                                                                                                                                                                                     <collection_shortname>_All


                                                                                                                                                                                                                                                                                                                                                                                                                                         addressed by “ned”
                                                                                                          Data Products
                                                                                                                                                                                                                <collection_shortname>_All


          •    HDF5 is a general purpose library and file format for                                                                                                                                                                                                                  1
                                                                                                                                                                                                                                                                                                                                                 <<HDF5 Dataset>>
                                                                                                                                                                                                                                                                                                                                                   <<HDF5 Dataset>>
                                                                                                                                                                                                                                                                                                                                                    <<HDF5 Dataset>>
                                                                                                                                                                                                                                                                                                                                                                                     •attributes




                                                                                                                                                                                                                                                                                                                                                                                                                                      – Are NPOESS products enough consistent, one to
                                                                                                                                                                                                                                                                                                                                                         <field>
                                                                                                                                                                                                                                                                                                                                                      <<HDF5 Dataset>>                             1
                                                                                                                 1                                                                                                                                                                                                                                     <<HDF5 Dataset>>
                                                                                                                                                                                                                                                                                                                                                          <field>

               storing scientific data. Two primary objects:                                                                                      <<HDF5 Dataset>>
                                                                                                                                                                                                                             1                                                                                 <<HDF5 Dataset>>
                                                                                                                                                                                                                                                                                                           <collection_shortname>_Ag
                                                                                                                                                                                                                                                                                                                                              •attributes
                                                                                                                                                                                                                                                                                                                                                •attributes
                                                                                                                                                                                                                                                                                                                                                            <field>
                                                                                                                                                                                                                                                                                                                                                             <field>
                                                                                                                                                                                                                                                                                                                                             •attributes <QA field>




                • Dataset, a multidimensional array of data elements
                                                                                                                                    1
                                                                                                                                               <collection_shortname>_
                                                                                                                                                           Aggr                                                                                            ned                                       1

                                                                                                                                                                                                                                                                                                           •attributes
                                                                                                                                                                                                                                                                                                                        gr
                                                                                                                                                                                                                                                                                                                                                 •attributes
                                                                                                                                                                                                                                                                                                                                                    •attributes
                                                                                                                                                                                                                                                                                                                                                    dataset of data
                                                                                                                                                                                                                                                                                                                                                      dataset of data
                                                                                                                                                                                                                                                                                                                                                       dataset of data
                                                                                                                                                                                                                                                                                                                                                         dataset of data
                                                                                                                                                                                                                                                                                                                                                          dataset of data
                                                                                                                                                                                                                                                                                                                                                                                                                                         another that a general purpose reader is possible?
                                                                                                                                             •attributes



                • Group, a structure for organizing objects
                                                                                                                                                                                                                                                                                                                                         1


                                                                                                                     *
                                                                                                                                             dataset of Object References 1
                                                                                                                                                     [one per field]
                                                                                                                                                                                                                                                                            <<HDF5 Group>>
                                                                                                                                                                                                                                                                                          *

                                                                                                                                                                                                                                                                                                 1
                                                                                                                                                                                                                                                                                                                dataset of Object
                                                                                                                                                                                                                                                                                                            References [one per field]
                                                                                                                                                                                                                                                                                                                                                                                                                                      – Can a computer program make detailed
          •
                                                                                                                                1

               Efficient storage and I/O, including parallel I/O.                                       <<HDF5 Group>>
                                                                                                    <collection_shortname>
                                                                                                                                                                                     {nf}
                                                                                                                                                                                            <<HDF5 Dataset>>
                                                                                                                                                                                                 <field>
                                                                                                                                                                                                                      {nf}
                                                                                                                                                                                                                                                                        <collection_shortname>

                                                                                                                                                                                                                                                                                                 1                                                    {nf}
                                                                                                                                                                                                                                                                                                                                                                 <<HDF5 Dataset>>
                                                                                                                                                                                                                                                                                                                                                                       <field>
                                                                                                                                                                                                                                                                                                                                                                                            {nf}

                                                                                                                                                                                                                                                                                                                                                                                                                                         associations between the HDF5 file and the XML
          •
                                                                                                                                1                                                                                                                                      •attributes

               Free, open source software, multiple platforms.                                     •attributes
                                                                                                                                                  <<HDF5 Dataset>>
                                                                                                                                               <collection_shortname>_
                                                                                                                                                                                     {nf}                             1
                                                                                                                                                                                                                                                                                                         ng
                                                                                                                                                                                                                                                                                                                <<HDF5 Dataset>>
                                                                                                                                                                                                                                                                                                            <collection_shortname>_Gra                {nf}
                                                                                                                                                                                                                                                                                                                                                             •attributes
                                                                                                                                                                                                                                                                                                                                                                                                                                         product profile?
          •
                                                                                                                                                                                                                                                                                                                         n_n

               Data stored in HDF5 is used in many fields from
                                                                                                                                        ng


                                                                                                                                                                                                                                                                                                                                                                                                                                      – Can the product profile be used to drive automated
                                                                                                                                                           Gran_n                                                                                                                                                                                                  dataset of data
                                                                                                                                                                                              dataset of data                                                                                               •attributes
                                                                                                                                             •attributes

                                                                                                                                                                                                                                                                                                                                         Attribute
               computational fluid dynamics to film making.                                                                                       dataset of Region
                                                                                                                                               References (Hyperslabs)
                                                                                                                                                    [one per field]
                                                                                                                                                                          1
                                                                                                                                                                                                                                                                                                                dataset of Region
                                                                                                                                                                                                                                                                                                             References (Hyperslabs)
                                                                                                                                                                                                                                                                                                                  [one per field]
                                                                                                                                                                                                                                                                                                                                     {nf}


                                                                                                                                                                                                                                                                                                                                         Aggregation …                                                                                   parsing of quality flags?
          •    Data can be stored in HDF5 in an endless variety of
                                                                                                                                                                                                                                                                                                                                                                                                                              •      To demonstrate the feasibility of implementation of
               ways, so it is important to standardize how NPOESS
                                                                                                                                                                                                                                                                                                                                                                                                                                     suggested enhancements in the self-description of
               product data is organized in HDF5.                                                                                                                                                                         XML Product Profile
                                                                                                  Current Information Model
                                                                                                                                                                                                                                                                                                                                                                                                                                     NPOESS products.
                                                                                                                                                                                                                                         1


                                                                                                  as delivered by NPOESS IDPS.
                                                                                                                                                                                                                                                       <field>

                                                                                                                                                                                                                                                                 Extract from XML Profile source …                                                                                                        HDF5                        – “ned” makes enhancements to the NPOESS format
                                                                                                  HDF5 file and associated XML                                                                                                               •attributes
                                                                                                                                                                                                                                                                                                                                                                                                       Enhanced                          for challenge items marked by 
                           Format Strengths                                                       Product Profile                                                                                                                                                                                                                                                                                       by “ned”

                                                                                                                                                                                                                                                                                                                                                                                                                                                    ned version 0.7
          •    Straight HDF5.
                 No need for additional libraries.                                                                                                                                                                                                                                                                                                                                                                           •      ANSI C code                                     •      Written against
          •    Consistent HDF5 group structure                                                                                                                                                                                                                                                                                                                                                                                         – HDF hdf5-1.6.5                                     NPOESS IDPS build 1.4
                 Organization for each product is the same as all                                          NED ….
                                                                                                                open attribute definition configuration file                                                                                                                                                                                                                                                                           – libxml 2.6.16                                      sample data and
                   others.                                                                                      open the input HDF5 file                                                                                                                                                                                                                                                                                                                                                    profiles.
                                                                                                                                                                                                                                                                                                                                                                                                                                       – udunits 1.4
                 Data “payload” is always in a product group within                                            create an output HDF5 file (make the "All_Data" and "Data_Product" paths)
                                                                                                                                                                                                                                                                                                                                                                                                                              •      Development                                     •      Numerous discrepancies
                   All_Data group.                                                                              for each product in the "Data_Products" path of the input HDF5 file (H5Giterate) {
                                                                                                                     Find and open the associated XML Product Profile                                                                                                                                                                                                                                                                    –     Apple Xcode                                  between profiles and
          •    Allows for flexible temporal aggregation
                                                                                                                     create a product group in the "Data_Products" path of the output HDF5 file                                                                                                                                                                                                                                          –     GNU C 4.0                                    sample data resolved by
                • Granules are appended by extending dataset
                                                                                                                     for the aggregate reference in the "Data_Products" path of the input HDF5 file {                                                                                                                                                                                                                         •      SLOCCount..                                            editing XML profiles (not
                   dimension.                                                                                              create an "aggregate reference stub" in the output HDF5 file                                                                                                                                                                                                                                                  –     ansic=3707                                   affecting build 1.4 profile
                                                                                                                              for each attribute attached to the aggregate (H5Aiterate)
                                                                                                                                                                                                                                                                                                                                                                                                                                         –     XML=1500                                     DTD).
                                                                                                                                         validate type and range
                          Format Challenges                                                                                             attach attribute to the aggregate reference stub of the output HDF5 file                                                                                                                                                                                                              •      Submitted for NASA                              •      After editing profiles, ned
                                                                                                                                        hold aggregate metadata for later                                                                                                                                                                                                                                                            technology transfer open                               successfully parses and
                                                                                                                     }                                                                                                                                                                                                                                                                                                               source release.                                        acts upon 50 of 53 non-
                                                                                                                     for each granule reference in the "Data_Products" path of the input HDF5 file {
          •    Geolocation appears in a separate product group and                                                                                                                                                                                                                                                                                                                                                                                                                          RDR sample files.
                                                                                                                              create a ”granule reference stub" in the output HDF5 file
               may be in separate HDF5 file.                                                                                  for each attribute attached to the granule (H5Aiterate)                                                                                                                                                                                                                                                                                                •      Not complete!
          •   Field metadata, used to interpret data (similar to netCDF                                                                validate type and range
                                                                                                                                        attach attribute to the granule reference stub of the output HDF5 file
               CF) are in separate product profile file.                                                                      "aggregate" the metadata according to attribute definition file (hold for later)
          •   Quality flags must be parsed before they can be                                                                                                                                                                                                                                                                                                                                                                What does ned do?                                      What does (0.7)not do?
                                                                                                                       }
                                                                                                                                                                                                                                                                                                                                                                                                                              •      Check the profile against the                   •      Copy some of the attributes.
               interpreted.                                                                                          create a product group in the "All_Data" path of the output HDF5 file                                                                                                                                                                                                                                           HDF5 product.
              Information needed for un-scaling scaled integers is not                                                                                                                                                                                                                                                                                                                                                                                                              •      Aggregate g-Rings.
          •                                                                                                            attach the aggregate metadata as HDF5 attributes of the product group in the "All_Data" path of output file.
                                                                                                                                                                                                                                                                                                                                                                                                                              •      Validate attributes in HDF5
                                                                                                                     attach the aggregated granule metadata as HDF5 attributes of the product group in the "All_Data" path of output file.                                                                                                                                                                                                                                           •      Aggregate percent attributes.
               obvious.                                                                                                                                                                                                                                                                                                                                                                                                              product (type and value).                       •      Compute reference regions.
          •   HDF5 indirect reference link API, used to link metadata to                                                                                                                                                                                                                                                                                                                                                     •      For each field, read 16                         •
                                                                                                                                        for each field in the associated product group in the "All_Data" path of the input HDF5 file {                                                                                                                                                                                               attributes from product profile
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Produce a user block.
               the data in NPOESS’ use is complex and not supported                                                                          Find and open the associated "sub-tree" of the XML profile.                                                                                                                                                                                                                                                                             •      Discover shared dimensions
                                                                                                                                                                                                                                                                                                                                                                                                                                     and attach them as HDF
               by all analysis COTS implementations.                                                                                         read metadata from XML profile and validate type and range                                                                                                                                                                                                                              attributes of the field dataset.                       and create appropriate
                                                                                                                                             ** "aggregate" the field metadata to associate with the data aggregation                                                                                                                                                                                                                                                                       aggregate attributes.
                                                                                                                                                                                                                                                                                                                                                                                                                              •      Attach scale and offset values
                                                                                                                                             for each NPOESS "datum" in the field {                                                                                                                                                                                                                                                  as “Scale_Factor” and
                                                                                                                                                   create a data aggregation field in the output HDF5 file                                                                                                                                                                                                                           “Add_Offset” attributes to field                Additional thoughts
               Information Model UML Diagram                                                                                                       copy the data from the input HDF5 file to the output HDF5 file                                                                                                                                                                                                                    dataset.                                        •      Modifications will be
                                                                                                                                                   attach the field metadata to the data aggregation field of the output HDF5 file                                                                                                                                                                                            •      Parse bit-wise fields and                              necessary to accommodate
                                                                                                                                             }                                                                                                                                                                                                                                                                                       create separate compressed                             build 1.5 product profiles.
                                                                                                                                        }                                                                                                                                                                                                                                                                                            datasets for each quality flag.                 •      Should the field attribute
                                                                                                                                                                                                                                                                                                                                                                                                                              •      Aggregate granule attributes                           names follow the CF
                                                                                                                                        ** for each field in the "All_Data" path of the output HDF5 file                                                                                                                                                                                                                             and attach them as attributes                          conventions?
                                                                                                                                              create the object reference to the field                                                                                                                                                                                                                                               of All_data product group.
                                                                                                                                              populate the "aggregate reference stub” in the "Data_Products" path of the output HDF5 file
                                                                                                                                              ** for each granule
                                                                                                                                                    for each field in the "All_Data" path of the output HDF5 file
                                                                                                                                                         create the region reference to the portion of the field appropriate to this granule
                                                                                                                                                         populate the "granule reference stub"in the "Data_Products" path of the output HDF5 file                                                                                                                                                                          Attributes extracted from XML Profile,
                                                                                                                            }
                                                                                                                                                                                                                                                                                                                                                                                                                       Inserted into HDF as attributes of dataset field
                                                                                                                                                                                                                                                                                                                                                                                                                   Attribute Name                    Type            Comments
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Data type is the type of the resulting unscaled datum. To un-scale the elements,
                                                                                                                                                                                                                                                                                                                                                                                                                   Add_Offset                        Number          first multiply the scaled element by the Scale_Factor and then add the Add_Offset.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     If the dataset is not scaled, the Add_Offset will exist and have a value of zero.

                                                                                                                                                                                            „Enhanced” UML Diagram                                                                                                                                                                                                                                                   String format is:
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     “%d-bit %s”, where %d is the number of bits and %s is one of:
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     o               signed integer
                                                                                                                                                                                                                                                                                                                                                                                                                   DataType                          String
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     o               unsigned integer
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     o               floating point
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     o               <blank> (number of significant bits in bitfield)
                                                                                                                                                                                                                                                                                                                                                                                                                   Description                       String          A descriptive text
                                                                                                                                                                                                                                                                                                                                                                                                                   MeasurementUnits                  String          Consistent with SI naming and Unidata’s “udunits” package
                                                                                                                                                                                                                                                                                                                                                                                                                   NumberOfFillValues                Integer         If zero, then no FillValue_Name and FillValue_Value attributes are attached.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     If zero, then no LegendEntry_Name and LegendEntry_Value attributes are
                                                                                                                                                                                                                                                                                                                                                                                                                   NumberOfLegendEntries             Integer
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     attached. Legend entries are used for quality fields only.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Maximum expected value of field elements in the product, not just this dataset
                                                                                                                                                                                                                                                                                                                                                                                                                   RangeMax                          Number
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     instance. Data type matches type of dataset.

                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Minimum expected value of field elements in the product, not just this instance.
                                                                                                                                                                                                                                                                                                                                                                                                                   RangeMin                          Number
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Data type matches type of dataset.

                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Data type is the type of the resulting unscaled datum. To un-scale the elements,
                                                                                                                                                                                                                                                                                                                                                                                                                   Scale_Factor                      Number          first multiply the scaled element by the Scale_Factor and then add the Add_Offset.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     If the dataset is not scaled, Scale_Factor will exist and have a value of one.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     True indicates that Scale_Factor and Add_Offset should be applied to the data to
                                                                                                                                                                                                                                                                                                                                                                                                                   Scaled                            Boolean         recover the un-scaled element values. Note that fill values are in the dataset type
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     and so must be tested before un-scaling.
                                                                                                                                                                                                                                                                                                                                                                                                                   FillValue_Name                    Set of string
                                                                                                                                                                                                                                                                                                                                                                                                                                                     Set of
                                                                                                                                                                                                                                                                                                                                                                                                                   FillValue_Value                                   Data type matches type of dataset.
                                                                                                                                                                                                                                                                                                                                                                                                                                                     number
                                                                                                                                                                                                                                                                                                                                                                                                                                                     Set of
                                                                                                                                                                                                                                                                                                                                                                                                                   Dimension_GranuleBoundary                         True (1) indicates that this is dimension extends when granules are appended.
                                                                                                                                                                                                                                                                                                                                                                                                                                                     Boolean
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Name match indicates that this dimension is congruent with the same dimension
                                                                                                                                                                                                                                                                                                                                                                                                                   Dimension_Name                    set of string
                                                                                                                                                                                                                                                                                                                                                                                                                                                                     names in other datasets in this product group.
                                                                                                                                                                                                                                                                                                                                                                                                                   Field_Name                        String          Potentially a place for the CF standard_name
                                                                                                                                                                                                                                                                                                                                                                                                                   LegendEntry_Name                  Set of string
                                                                                                                                                                                                                                                                                                                                                                                                                                                     Set of
                                                                                                                                                                                                                                                                                                                                                                                                                   LegendEntry_Value                                 Data type matches type of dataset.
                                                                                                                                                                                                                                                                                                                                                                                                                                                     number
                                                                                                                                                                                                                                                                                                                                                                                                                   NumberOfDimensions                Integer         Integer greater than zero. This duplicates the HDF attribute.




NASA • NPP                                      SCIENCE DATA SEGMENT • IPO• ALGORITHM                                                                                                                                                                                                                                                                                                                                                                                   DIVISION
Richard E. Ullman             NASA/GSFC/NPP • NOAA/NESDIS/IPO               Data / Information Architecture • Algorithm / System Engineering                                                                                                                     richard.e.ullman@nasa.gov

				
DOCUMENT INFO
Shared By:
Categories:
Stats:
views:9
posted:4/19/2010
language:Indonesian
pages:1