Docstoc

D ICF Concepts TN

Document Sample
D ICF Concepts TN Powered By Docstoc
					                                                reference GAEL-ICF-TN-060.2

    GAEL                                           issue 2 revision 3
                                                    date 13/04/2000
Consultant
                                                   page 1 of 32




                             Development of a Baseband
                               Data Archive Interchange
                                                 Format
                                ICF Concept Technical Note




                             name
                             function            date
                             company             signature



              prepared by    Serge RIAZANOFF
                             project manager
                             Gael Consultant



              approved by




             authorized by   Gian Maria PINNA
                             project manager
                             ESA – ESRIN
                         Development of a baseband data archive                 reference GAEL-ICF-TN-060.2

        GAEL                                 interchange format                    issue 2 revision 3
                                                                                     date 13/04/2000
    Consultant                                  ICF Concepts technical note
                                                                                     page 2 of 30




Document Status Sheet
    Issue   Date         Comments                                                         Author



     1.0    22/10/1999   Issue as formal „Development of a Baseband Data Archive          A. Mbaye
                         InterChange Format‟ project deliverable

     2.0    25/01/2000   Decisions taken during the meeting held at ESRIN the             S. Riazanoff
                         26/10/1999.

     2.1    10/03/2000   Decisions taken during the meeting held at Gael Consultant the S. Riazanoff
                         10/03/2000.

     2.2    27/03/2000   Decisions taken by e-mail exchange on 27/03/2000. Version        S. Riazanoff
                         sent to ATT members before meeting London April 2000.

     2.3    13/04/2000   Review of document during CEOS WGISS meeting held at             S. Riazanoff
                         London 11-14 April 2000.
                                                Development of a baseband data archive                                                reference GAEL-ICF-TN-060.2

                  GAEL                                              interchange format                                                     issue 2 revision 3
                                                                                                                                            date 13/04/2000
              Consultant                                                              ICF Concepts technical note
                                                                                                                                           page 3 of 30




Contents




DOCUMENT STATUS SHEET........................................................................................................................... 2

CONTENTS ........................................................................................................................................................... 3

1       INTRODUCTION ......................................................................................................................................... 5
     1.1     SCOPE AND PURPOSES .............................................................................................................................. 5
        1.1.1. Rationale ............................................................................................................................................... 5
        1.1.2. Place of this document in the project .................................................................................................... 5
        1.1.3. Archive Format Technical Note............................................................................................................. 6
        1.1.4. Basic Principles ..................................................................................................................................... 6
        1.1.5. Archive and InterChange Formats ........................................................................................................ 6
        1.1.6. Limitations ............................................................................................................................................. 6
        1.1.7. Document Overview .............................................................................................................................. 6
     1.2     APPLICABLE AND REFERENCE DOCUMENTS ............................................................................................. 7
        1.2.1. Applicable documents ............................................................................................................................ 7
        1.2.2. Reference documents ............................................................................................................................. 7
        1.2.3. Reference documents – Archive formats ................................................................................................ 7
        1.2.4. Reference documents – Information description methodology .............................................................. 8
     1.3     ABBREVIATIONS AND ACRONYMS ............................................................................................................ 8
     1.4     DEFINITIONS........................................................................................................................................... 10
2       CEOS ICF CONCEPTS.............................................................................................................................. 11
     2.1. INTERCHANGE CONTEXT ........................................................................................................................ 11
        2.1.1. From Earth Observation to end-users ................................................................................................. 11
        2.1.2. Mission-dependent exchanges ............................................................................................................. 12
        2.1.3. Telemetry processing levels ................................................................................................................. 12
        2.1.4. Data formats and configuration control .............................................................................................. 12
     2.2. DATA EXCHANGE SCENARII ................................................................................................................... 14
        2.2.1.     Interchange Driven by Export Facility .......................................................................................... 14
        2.2.2.     Interchange Driven by Import Facility .......................................................................................... 15
3       CEOS ICF HIGH LEVEL REQUIREMENTS ........................................................................................ 17
     3.1. DIFFERENCES WITH ARCHIVING REQUIREMENTS .................................................................................... 17
        3.1.1. Memory constraints ............................................................................................................................. 17
        3.1.2. Run-time constraints ............................................................................................................................ 17
        3.1.3. Strong portability constraint ............................................................................................................... 17
        3.1.4. Released physical reliability ................................................................................................................ 17
        3.1.5. Direct access to archive ...................................................................................................................... 17
        3.1.6. External references .............................................................................................................................. 18
        3.1.7. Impact of software changes ................................................................................................................. 18
        3.1.8. Sensor Description for Earth Observing Dynamic .............................................................................. 18
        3.1.9. Open Archival Information System ...................................................................................................... 18
     3.2. HIGH LEVEL REQUIREMENTS .................................................................................................................. 19
4.      CEOS ICF DETAILED REQUIREMENTS ............................................................................................. 20
     4.1.       BASEBAND INTERCHANGE UNIT (BIU) .................................................................................................. 20
                                             Development of a baseband data archive                                                reference GAEL-ICF-TN-060.2

               GAEL                                              interchange format                                                     issue 2 revision 3
                                                                                                                                         date 13/04/2000
           Consultant                                                              ICF Concepts technical note
                                                                                                                                        page 4 of 30




  4.2. BASEBAND INTERCHANGE MEDIUM ....................................................................................................... 20
     4.2.1. Multi-media exchange ......................................................................................................................... 20
     4.2.2. File naming .......................................................................................................................................... 20
     4.2.3. Use of tapes and data packaging ......................................................................................................... 20
     4.2.4. Independence from medium and limitations of operating system ........................................................ 20
     4.2.5. Data segmentation ............................................................................................................................... 21
  4.3. BIU ORGANIZATION............................................................................................................................... 21
     4.3.1. File structure ....................................................................................................................................... 21
     4.3.2. File order ............................................................................................................................................. 21
  4.4. METADATA MANAGEMENT .................................................................................................................... 21
     4.4.1. Metadata Organization........................................................................................................................ 21
     4.4.2. Language for Metadata Description ................................................................................................... 21
     4.4.3. DED organization ............................................................................................................................... 22
     4.4.4. Mission-dependent section .................................................................................................................. 23
  4.5. SIGNAL DATA MANAGEMENT ................................................................................................................ 24
     4.5.1. No redundancy..................................................................................................................................... 24
     4.5.2. No encryption ...................................................................................................................................... 24
     4.5.3. CRC and Checksum ............................................................................................................................. 24
     4.5.4. No compression ................................................................................................................................... 24
     4.5.5. PN decoded .......................................................................................................................................... 24
     4.5.6. Data is frame synchronized ................................................................................................................. 24
     4.5.7. Chronological order ............................................................................................................................ 24
     4.5.8. Byte alignment ..................................................................................................................................... 24
     4.5.9. Signal data record ............................................................................................................................... 25
  4.6. CEOS ICF VERSION MANAGEMENT ...................................................................................................... 25
     4.6.1. Minor/major versions .......................................................................................................................... 25
     4.6.2. DED and software availability ............................................................................................................ 25
APPENDIX A – BASEBAND DATA CONCEPT COMPLIANCE MATRIX .............................................. 26
  A.1.       INTRODUCTION ....................................................................................................................................... 26
  A.2.       COMPLIANCE MATRIX ............................................................................................................................ 26
APPENDIX B – CEOS ICF DATA ENTITY DICTIONARY ......................................................................... 28
  B1.        INTRODUCTION ....................................................................................................................................... 28
  B.2.       HIERARCHICAL REPRESENTATION .......................................................................................................... 28
                                  Development of a baseband data archive                   reference GAEL-ICF-TN-060.2

            GAEL                                      interchange format                      issue 2 revision 3
                                                                                               date 13/04/2000
        Consultant                                         ICF Concepts technical note
                                                                                              page 5 of 30




1      INTRODUCTION
1.1    Scope and Purposes
       This document serves as a description of the Baseband Data InterChange Format Concepts for the
       Committee on Earth Observation Satellites (CEOS). It has been reviewed by the Archive Task Team of
       the CEOS Working Group on Information Systems and Services (WGISS) Data during the ATT Meeting
       held at London, April 2000.

       1.1.1. Rationale
       The development of a Baseband Data Archive Interchange Format (CEOS ICF) comes from a real need to
       exchange data between different archiving facilities, which do not employ the same processing systems.
       CEOS ICF shall be an intermediate format that all archive centers can interpret. Facilities should translate
       their own format into CEOS ICF to export their archive data and shall translate CEOS ICF into their own
       format in order to import other facility data (Figure 1.a).




           Export Facility                                                                  Import Facility


               Archive                              CEOS ICF                           Archive
               Format                                 Format exchange through CEOS ICF Format
                             Figure 1.a: Representation of data

       1.1.2. Place of this document in the project

    « Archive Formats Technical Note » document                      Archive Format
                                                                       document
                                                                      Archive Format              Archive
                                                                        documents
                                                                        Archive Format             Data
                                                                         FRED
         « ICF Concepts Technical Note » document                          document
                                                                          Archive Format          FRED
                                                                          MDPS                    MDPS
         - High level requirements                                           document
                                                                           GERALD
              - Detailed requirements
                                                                             VEXCEL
                   - Data Entity Dictionary (Entries)
                                                                                                 Metadata
                                                                                                Description
                                                                                                 Information
                                                                                                Methodology
                                                                                                   Information
                                                                                                 Description
                          « CEOS ICF » document                                                    Description
                                                                                                 Methodology
                                                                                                     Information
                                                                                                 DEDSL
                             - Format description                                                 Methodology
                                                                                                     Description
                                                                                                     PVL
                                                                                                    Methodology
                             - Data Entity Dictionary (Full description)                               ODL
                                                                                                        XML

                                            ICF Translator demonstration software
       Deliverable of this project

                                             Figure 1.b: Documents roadmap.
                          Development of a baseband data archive                    reference GAEL-ICF-TN-060.2

     GAEL                                     interchange format                          issue 2 revision 3
                                                                                          date 13/04/2000
 Consultant                                        ICF Concepts technical note
                                                                                          page 6 of 30




1.1.3. Archive Format Technical Note
The feasibility of such interchange format was assessed through a preliminary analysis (R-1). This last
was undergone within the same current project. It showed that:
- information commonalties between existing archive formats are enough to allow a transfer of a
consistent amount of data,
- differences of structure and syntax do not prevent the information exchange. Obviously, some losses are
possible but storing extra information in separated structures should overcome most of them.
Moreover, this preliminary analysis highlighted the fact that it is possible to complete efficiently a target
format as long as not only source format is employed but also all necessary external sources of
information. Indeed, the exchange does not concern solely archive formats, it involves the source and
target whole information systems.

1.1.4. Basic Principles
The basic principles of CEOS ICF may be expressed with the following points:
- it shall settle a strong formalism for archive metadata which shall be clearly understood, easily
employed and hopefully widely accepted by the community,
- it shall also provide a flexible and appropriate structure which allow data transfer as fast (in terms of
development time) and inexpensive as possible,
- finally, it shall foresee and favor the development of powerful translation software.
These points shall be developed in the followings.

1.1.5. Archive and InterChange Formats
The importance of data exchange has been clearly underlined within the “Baseband Data Concept”
document. This document was already considering possible differences between goals of data archive and
data exchange, these differences will be more detailed within section 2.
In the same time, strong requirements of the “Baseband Data Concept” aimed to preserve the original data
will lead to state high level requirements.

1.1.6. Limitations
Even if the proposed hierarchical data organization could allow considering other type of data, the present
analysis only concerns 2-Dimensional images that represent the largest amount of data presently
processed by Earth Observation missions.

1.1.7. Document Overview
Scope of this document is to provide the ATT meeting members with the maximum material allowing to
assess high level principles with regard to their direct implications in data organization. The next
document that will detail “CEOS InterChange Format” it-self will be a strict application of the detailed
requirements of the present document after having been released by the ATT meeting.
“1. INTRODUCTION” is this section. It will contain a detailed list of acronyms and definitions that
should try to realize a consensus within the user community because they will be used when defining the
CEOS ICF Data Entity Dictionary.
“2. CEOS ICF CONCEPTS”. This section summarizes the context in that interchange takes place. The
first part of this section describes the acquisition, archiving and production interdependencies. The second
part analyzes the procedures for setting and controlling definitions by the groups having the necessary
authority. The last part presents two philosophies that could be adopted for baseband data interchange.
                              Development of a baseband data archive                  reference GAEL-ICF-TN-060.2

           GAEL                                   interchange format                      issue 2 revision 3
                                                                                           date 13/04/2000
       Consultant                                      ICF Concepts technical note
                                                                                          page 7 of 30




      “3. CEOS ICF HIGH LEVEL REQUIREMENTS”. After having underlined the main differences between
      the requirements dedicated to baseband data archiving and the ones dedicated to interchange of these
      data, this section lists a series of high level requirements that should govern exchange of baseband data
      archive.
      “4. CEOS ICF DETAILED REQUIREMENTS”. High level considerations listed in the previous section
      will generate more detailed requirements. The final “CEOS ICF” document will be a straightforward
      implementation of the detailed requirements laid down in this section.
      “APPENDIX A – BASEBAND DATA CONCEPT COMPLIANCE MATRIX”. Compliance with
      Baseband Data Concepts is the highest recommendation for ICF development. A compliance matrix will
      guaranty that detailed requirements developed in section 4 fulfill this adherence.
      “APPENDIX B – CEOS ICF DATA ENTITY DICTIONARY”. This section presents the first
      hierarchical organization of data entities that will be kept in metadata file.



1.2   Applicable and Reference Documents

      1.2.1. Applicable documents
      A-1       AEF99/002/GMP - Development of a Baseband Data InterChange Format v1.0 -
                ESA/ESRIN - 7 June 1999.
      A-2       Baseband Data Concept White Paper - Draft Version - 06 May 1999 - ATT CEOS WGISS.
      A-3       GAEL-ICF-MM-060.4 - Minutes of Meeting - ESRIN - 26/10/1999.
      A-4       GAEL-ICF-MM-060.5 - Minutes of Meeting - GAEL Consultant – 10/03/2000.



      1.2.2. Reference documents
      R-1       GAEL-ICF-TN-060.1 - Archive Format Technical Note - Issue 1, Version 2 – 10/03/2000 -
                GAEL Consultant.
      R-2       CCSDS 100.0-G-1 – Telemetry – Summary of Concept and Rationale – CCSDS – December
                1987.



      1.2.3. Reference documents – Archive formats
      F-1       DG-MA-50-6897 – Framed Raw Expanded data Product Specification, Issue/Rev. 2/10 –
                MDA – 30 Oct. 1998.
      F-2       C447-ST-73-103-CN – SPOT Archive GERALD Format Specification, Ed.1 Rev.0 –
                27/10/1997 - CNES.
      F-3       MDPS Systems – Multi Satellite data Processing Systems – Transcribed Data Format on
                DLT – Release 4.1.1. – 11 February 1999 - ACS.
      F-4       VX-STF-001 – Vexcel level 0 Processor (SKY) – Telemetry data Format v.1.0 – 26/07/1999 –
                VEXCEL Corporation.
      F-5       Landsat 7 System – Data Format Control Book (DFCB) Volume IV – Wideband Data –
                Revision L – 11 June 1999 – NASA/GSFC.
      F-6       S-IF-O/E-10-SI – SPOT to Direct Receiving Station Interface Description – 01/12/1997 –
                SPOT IMAGE.
                             Development of a baseband data archive                  reference GAEL-ICF-TN-060.2

           GAEL                                  interchange format                      issue 2 revision 3
                                                                                         date 13/04/2000
       Consultant                                     ICF Concepts technical note
                                                                                         page 8 of 30




      F-7       ER-IS-ESA-GS-002 – ERS-2 to Ground Segment Interface Specification, Issue 1.1 -
                02.12.1993 – ESA/ESTEC.



      1.2.4. Reference documents – Information description methodology
      M-1       CCSDS 647.0-R-2.6 - Data Entity Dictionary Specification Language (DEDSL) Abstract
                Syntax - October 1999 – CCSDS.
      M-2       FGDC-STD-001-1998 - Content Standard for Digital Geospatial Metadata - Federal
                Geographic Committee – June 1998.
      M-3       FGDC-STD-???-???? - Content Standard for Digital Geospatial Metadata: Extensions for
                Remote Sensing Metadata - Federal Geographic Committee – Internal Committee Draft,
                version 0.2 – September 8, 1999.
      M-4       Draft, A Strawman Proposal for a Standard Sensor Description Format of Earth Observing
                Dynamic Sensors - 20.09.1998 – Mike BOTTS, prepared for CEOS WGISS.
      M-5       CCSDS 650.0-R-1 - Reference Model for an Open Archival Information System (OAIS) - June
                1999 - CCSDS.
      M-6       CEOS.WGISS.DS.TN01 - Guidelines on Standard Formats and Data Description Languages
                Version C Draft - 03.09.1997 - CEOS WGISS Guidelines.
      M-7       Alain MICHARD – XML Language et applications - Eyrolles – 1999.
      M-8       St Laurent, Cerami - Building XML Applications - Mc Graw-Hill – 1999.
      M-9       Michael MORRISON, et al. – XML Unleashed – Sams Publishing – 2000.
      M-10      Planetary data System Standards Reference – Jet Propulsion laboratory                         –
                URL: http://pds.jpl.nasa.gov/stdref – Updated 10/11/95 – Browsed 24/02/2000.
      M-11      Robert Orfali, Dan Harkey - Client/Server Programming with JAVA and CORBA - Wiley
                Computer publishing – 1998.




1.3   Abbreviations and Acronyms
      This list is a collection of terms found within the referenced documents. When an abbreviation has been
      found within only one document its unique reference is reported inside parenthesis.
      API              Application Programming Interface
      ASCII            American National Standards Institute
      ATT              Archive Task Team
      BER              Bit Error Rate
      BIL              Band Interleaved by Line
      BIP              Band Interleaved by Pixel
      BSQ              Band Sequential
      CADU             Channel Access Data Unit (delimited by Synchronization codes)
      CCD              Charged-Coupled Device
      CCF              Compatible Computer Format
      CCSDS            Consultative Committee for Space Data Systems
                     Development of a baseband data archive                 reference GAEL-ICF-TN-060.2

     GAEL                                interchange format                    issue 2 revision 3
                                                                                date 13/04/2000
 Consultant                                   ICF Concepts technical note
                                                                               page 9 of 30




CEOS          Committee on Earth Observation Satellite
CNES          Centre National d‟Etudes Spatiales
CONI (F-4)    Tagged ASCII File
CRC           Cyclic Redundancy Check
DCS (F-4)     Direct Capture System
DED           Data Entity Dictionary
DEDSL         Data Entity Dictionary Specification Language
DLT           Digital Linear Tape
DPCM          Differential Pulse Code Modulation
EAF           Existing Archive Format
ECC           Error Correcting Code
EO            Earth Observation
EOF           End Of File
EOR           End Of Record
EOT           End Of Tape
EROS          Earth Resource Observation Systems
ESA           European Space Agency
FTP           File Transfer Protocol
GB            GigaByte(s)
HDDT          High Density Data Tape
HR            High Rate (Data)
ICF           Baseband Data Archive InterChange Format
IDHT (F-8)    Instrument Data Handling on Transmission Subsystem
IEEE          Institute of Electrical and Electronic Engineers
MIR           Moyen Infra-Rouge
NASA          National Aeronautics and Space Agency
NASDA         National Space Development Agency of Japan
NORAD         North American Aerospace Defense Command
OBR           On-Board Recorder (or On-Board Recording (F-4))
ODL           Object Description Language
OGRC          On-Ground Range Compression
PCD           Payload Control Data
PN (or PRN)   Pseudorandom noise
PRF           Pulse Response Frequency
PVL           Parameter Value Language
SAR           Synthetic Aperture Radar
SPOT          Satellite Pour l‟Observation de la Terre
                              Development of a baseband data archive                   reference GAEL-ICF-TN-060.2

           GAEL                                   interchange format                      issue 2 revision 3
                                                                                           date 13/04/2000
       Consultant                                      ICF Concepts technical note
                                                                                          page 10 of 30




      SW               Software
      SWST             Sample Window Start Time
      TBC              To Be Confirmed
      TBD              To Be Defined
      URL              Uniform Resource Locator
      UTC              Universal Time Coordinated
      VCDU             Virtual Channel Data Unit
      WGISS            Working Group on Information Systems and Services
      WRS              World Reference System
      XML              eXtensible Markup Language




1.4   Definitions
      Many words or groups of words are found within the various documents with sometimes slightly different
      meaning. The following list of definitions will be used consistently throughout this document.
      format           Description of the way data are organized.
      frame            Telemetry unit.
      segment          Period of time in which sensor is operating in a single mode.
                               Development of a baseband data archive                            reference GAEL-ICF-TN-060.2

             GAEL                                  interchange format                                  issue 2 revision 3
                                                                                                       date 13/04/2000
         Consultant                                          ICF Concepts technical note
                                                                                                       page 11 of 30




2    CEOS ICF CONCEPTS
     Scope of this section is threefold:
          Analyze the present context of the various tasks going from the data acquisition to their distribution
           to end-users. Baseband data interchange is precisely located with regard to the other activities:
           acquisition, archiving and production.
          Identify the organizations having the knowledge and the authority to define the form and contents of
           the information involved in baseband data interchange.
          Set out two strategies that could be adopted for baseband data interchange: -a “interchange driven by
           export facility” and –a more modern “interchange driven by import facility”.



2.1. Interchange context
     This section addresses two of the major difficulties of baseband data interchange:
     1.    What are exactly the pre-processing levels of telemetry that are achieved in existing systems?
     2.    Who has authority to define and control the contents and meaning of the various data entities that can
           be found in existing archives and in the interchange format?

     2.1.1. From Earth Observation to end-users


                     Space segment
             OBR



          original signal         Transmission
               data                subsystem
                                                               Ground segment
           instruments

                               transmitted telemetry                           CEOS ICF
                                        Ground segment
                                                                    Export CEOS        Import CEOS
                                                                   ICF subsystem      ICF subsystem
                                                                               baseband
                                                                                archive
            Earth
                                   Receiving            received      Archiving           Production          end-user
                                   subsystem           telemetry      subsystem           subsystem           product




                                            Figure 2.1.a.: Data flow diagram.
     Original signal data          are the outputs of sensors when satellite processes acquisition within the
                                   reception area of the ground station or data played back from On-Board
                                   Recorder(s) (OBR).
     Transmitted telemetry         is the serial stream of bits than have been arranged in a transmission format.
                                   CCSDS has settled standards for this transmission like the organization in
                                   CADU‟s and transfer frames.
                         Development of a baseband data archive                    reference GAEL-ICF-TN-060.2

        GAEL                                 interchange format                        issue 2 revision 3
                                                                                       date 13/04/2000
    Consultant                                    ICF Concepts technical note
                                                                                       page 12 of 30




Received telemetry           Transmitted telemetry in bitstream format possibly corrupted during its
                             transfer and filtered by front-end sub-system.
Baseband archive             After having been pre-processed (see here below), telemetry is arranged in an
                             archive format (FRED, MDPS, GERALD, VEXCEL) and stored onto high-
                             density media in computer compatible format.
End-user product             Upon work ordering, data are extracted from the archive, processed and re-
                             arranged in an intermediate format (see level 0 of SPOT or Wideband of
                             Landsat 7) or a distribution format for end-users.

2.1.2. Mission-dependent exchanges
Some satellite operating agencies have already set-up an exchange policy to fulfill their proper needs.
 Direct Receiving Stations of the SPOT network shall be able to produce level 0 scenes in CAP (or
  CRIS few years ago) format. This product is very close the first distributed processing level (1A) that
  only performs radiometric corrections with respect to level 0.
     In this way, SPOT IMAGE may process scenes that have been received by other ground segments.
 For Landsat 7 mission, NASA has issued a Wideband data format (F-5) that defines the output of LPS
  (Landsat processing System). This format seems to be the one used to archive 0R data at the EROS
  Data Center. A note entitled “Landsat 7 Computer Compatible Raw Wideband exchange Data Format
  Control Book, version 1.0” has been issued by USG in June 1999 stating the way archives will be
  exchanged between the EROS Data Center and the international ground stations.
     At the contrary of SPOT, exchange concerns complete subintervals and not only single scenes.

2.1.3. Telemetry processing levels
Details of the way FRED, MDPS, GERALD and VEXCEL process telemetry before archiving is
analyzed in section “3.1. Validation of Image Data Exchange” of “Archive Format Technical Note”
(R-1).
This study applied to the three missions/instruments (Landsat ETM+, SPOT 1,2,3,4 XS,XI,P/M and ERS
SAR HR OGRC) shows that formats may be:
     very close: example of ERS SAR, where the raw telemetry is almost left unchanged by FRED,
      MDPS or VEXCEL, or
     very different: example of SPOT, where FRED keeps the original telemetry and GERALD pre-
      processes data until a level 0 format.
Nevertheless, all the formats process base pre-processing: -PN decoding, -downlinked channels are
interleaved, -play-back data are inverted, -Red Salomon corrections are applied, -DPCM is decompressed
and –data are almost always unpacked into bytes.
Even for the more elaborated level 0 format, changes seem to be limited to a re-arrangement of the
transmitted data and are reversible.



2.1.4. Data formats and configuration control
Scope of this section is to identify the various format types and the organizations in charge of controlling
them (see figure 2.1.b.).
                         Development of a baseband data archive                    reference GAEL-ICF-TN-060.2

        GAEL                                 interchange format                        issue 2 revision 3
                                                                                        date 13/04/2000
    Consultant                                     ICF Concepts technical note
                                                                                       page 13 of 30




                             Satellite Operating       Product Distribution
                             Agency                                Agency


                             Archive                                 Archive
                            Processor                               Processor
                            Designer 1                              Designer 2




      telemetry                archive                                archive                  end-user
        format                format 1          ICF format           format 2                   format



                                              CEOS WGISS
                                                    Standardization
                                                        bodies

                   Figure 2.1.b.: Data format and configuration control responsibilities.
Telemetry format      This format has been issued by the satellite operating agency (ESA for ERS, NASA
                      for Landsat, CNES for SPOT, NASDA for JERS…) and distributed to the ground
                      stations within a document often called “Satellite to Ground Station Interface
                      Specifications”. This reference document defines the way data are transmitted from
                      satellite and possibly the way auxiliary data useful for further processing are
                      transmitted from Mission Operation Center. This document is unique and contents
                      the exhaustive list of data entities that are handled by the mission.
Archive formats       These formats are normally designed by the companies, sometimes in conjunction
                      with the operating agency, that have developed complete systems able to receive,
                      archive and produce end-user data for one or more missions, or by the operating
                      agency. There is no standard to state at which processing level shall stay the archive
                      (for example, a raw telemetry may be archived in FRED instead data in GERALD
                      archive are close the end-user representation). CEOS has proposed the Baseband
                      Data Concepts as the baseline for the archive data.
End-user formats      In the majority of the cases, end-user format are defined by the satellite Operating
                      Agency it-self. But, in the same way distribution is sub-contracted to subsidiaries or
                      other organizations, end-user formats are sometimes defined by these third-part
                      organizations (example of SPOT IMAGE that produces and distributes images from
                      the satellite managed by the CNES). Because the tendency is to distribute added-
                      value products with always less auxiliary data, these end-user formats provide few
                      data entities to be kept in CEOS ICF. Only the knowledge of the way end-user
                      products are generated allow to determine which data entity shall be kept in CEOS
                      ICF to reach the same performances as for data directly received from the station.
CEOS ICF is developed and will be managed by CEOS WGISS. As shown before, it is essential to
maintain a close collaboration with the following experts:
     Mission experts – These people are the more qualified to determine the list of data entities (name,
      definition, default value, allowed values, mandatory/optional status…) that concern their mission.
                              Development of a baseband data archive                     reference GAEL-ICF-TN-060.2

            GAEL                                  interchange format                         issue 2 revision 3
                                                                                              date 13/04/2000
        Consultant                                      ICF Concepts technical note
                                                                                             page 14 of 30




         Archive format experts – These people can define which data are required in input of their archive to
          allow further processing and which information is required to ensure the long-term preservation of
          data.
         Standard experts – Designed from the standards, ICF format shall be maintained under a strict
          configuration control by an organization that will stay in close collaboration with the standardization
          bodies.



2.2. Data Exchange Scenarii
    The interchange operation may be seen in two manners that lead to two different implementation
    procedures and two types of application tools :
    -     interchange driven by export facility,
    -     interchange driven by import facility.

    2.2.1.      Interchange Driven by Export Facility
    The archiving facility develops a defined translator where available information is introduced into ICF
    version of its own dataset. The information may come from the archive format but also from external
    source. The goal of this task is to fill as much as possible the CEOS ICF format (Figure 2.2.1).
    When another facility requires those data, the exporting operations are :
    -     select the requested dataset and the respective data from respective auxiliary archive,
    -     convert the information into the CEOS ICF,
    -     dump resulting data onto a medium or transfer it through network.


    When the import facility receives the data, the importing operations are :
    -     convert the CEOS ICF dataset into the import format,
    -     record the resulting data set onto the archive medium,
    -     record the default value when required information is not available,
    -     keep interchange metadata file on a separated server.


    More sophisticated operations such as the following ones may be added :
    -     the export facility shall send first the current CEOS ICF DED for expertise. Comparison with
          equivalent files of the import facility shall lead to the exchange losses assessment before effective
          transfer ,
    -     it can undergo some reprocessing or some quality control of resulting data with the current archiving
          system,
    -     it can add to the interchange metadata file some data from CEOS ICF, some precision, some extra
          comment, data which were not recorded.
                           Development of a baseband data archive                  reference GAEL-ICF-TN-060.2

     GAEL                                      interchange format                      issue 2 revision 3
                                                                                        date 13/04/2000
 Consultant                                        ICF Concepts technical note
                                                                                       page 15 of 30




         Export Facility



          AUXILIARY                   ARCHIVE
            ARCHIVE                  PRODUCT
                                                                                 EXTERNAL
                                                                                  SOURCES

                       ICF WRITER




                            ICF
                            PRODUCT




                       ICF READER




          AUXILIARY                   ARCHIVE
            ARCHIVE                  PRODUCT                     OPERATOR



         Import Facility



                   Figure 2.2.1 : Representation of exchange driven by the export facility



2.2.2.     Interchange Driven by Import Facility
The idea of this exchange is to avoid transferring information which shall not be required by the import
facility. The export shall make a request according to CEOS ICF formalism of information it needs.
When available, import facility shall convert it into CEOS ICF formalism.
Those exchanges shall be implemented through Request Broker Architecture which will allow real time
transfer of information (M-112).
                        Development of a baseband data archive                   reference GAEL-ICF-TN-060.2

     GAEL                                   interchange format                       issue 2 revision 3
                                                                                      date 13/04/2000
 Consultant                                      ICF Concepts technical note
                                                                                     page 16 of 30




EXPORT FACILITY



AUXILIARY                 ARCHIVE                                                     EXTERNAL
                         PRODUCT                         OPERATOR                         SOURCES
 ARCHIVE



 ICF READER               ICF READER                                                      ICF READER




                                            Format Request Broker




  ICF WRITER                ICF WRITER




  FACILITY                  ARCHIVE
   SERVER                  PRODUCT


Import Facility



                  Figure 4.2 : Representation of exchange driven by the import facility
                              Development of a baseband data archive                    reference GAEL-ICF-TN-060.2

          GAEL                                    interchange format                        issue 2 revision 3
                                                                                            date 13/04/2000
      Consultant                                       ICF Concepts technical note
                                                                                            page 17 of 30




3    CEOS ICF HIGH LEVEL REQUIREMENTS
3.1. Differences with archiving requirements
     Scope of this section is to resume the major objective differences between requirements suitable for
     archiving from those suitable for interchange.

     3.1.1. Memory constraints
     Interchange will concern only a part of existing archives. Although archiving is a systematic process
     performed every day whatever being the quality of gigabytes of data, interchanges are often punctual
     activities that normally concern only small amount of data of particular interest.
     As a consequence, data of ICF shall not use compression techniques nor sophisticated physical
     organization on singular high-density support that are not foreseen in the external archive formats.

     3.1.2. Run-time constraints
     At present, archiving is processed in near real-time. In a next future, we may imagine that telemetry
     resulting from direct capture will be processed on-the-flight onto high-density media without intermediate
     storing on local disks. It can also be an off-line process performed during transcription operations but
     normally the format is kept the same as the real-time one, in order to ensure compatibility.
     Interchange is processed off-line and export facility has time to re-arrange data into a more convenient
     format. As a consequence, ICF may require more processing to possibly re-arrange and process baseband
     data archives.
     An other impact concerns the sequencing of data. Although some information (for example, the amount
     of data, their quality, the time interval…) may only be assessed once the complete acquisition is
     performed, CEOS ICF can compile a metadata file that will contain all this information and position this
     file before the data themselves for sequential access media (tapes or electronic links).

     3.1.3. Strong portability constraint
     Whereas the existing archives are often thought as internal storage unit within the acquisition and
     production system, CEOS ICF shall guaranty a perfect portability between heterogeneous systems.
     ICF shall be multi-media: tapes of whatever density, disk representation and electronic transfer.

     3.1.4. Released physical reliability
     Durability of acquired data is a critical issue for archiving. As shown in the study of existing archive
     formats (R-1), these systems use mechanisms (duplication of data, redundancy checking, periodical
     transcriptions) to guaranty such durability in front the aging of supports.
     At the contrary, CEOS ICF shall not consider this issue. The medium that will be used to exchange the
     data will be often occasional. If the data cannot be read from the destination system, exchange will be
     processed again.

     3.1.5. Direct access to archive
     Archive formats are almost always designed together with the production system that will extract the data
     to generate end-user products. Mechanisms are set up to allow easier and/or faster access to the archive
     from the production system.
     This issue is not a requirement for CEOS ICF.
                          Development of a baseband data archive                   reference GAEL-ICF-TN-060.2

        GAEL                                  interchange format                       issue 2 revision 3
                                                                                        date 13/04/2000
    Consultant                                     ICF Concepts technical note
                                                                                       page 18 of 30




3.1.6. External references
In almost all the studied archive formats (R-1), codes are used to identify some data entities (mission,
instrument, processing level, transmission mode…). These codes or mnemonics are understandable by the
production system and allows to spare memory space in archive.
Because CEOS ICF shall not consider memory constraints and shall be self-consistent, completeness, use
of external references (mnemonic tables, databases, external functions providing metadata…) shall be
avoided, in favor of a more explicit representation.

3.1.7. Impact of software changes
Archives have been designed for long term. Changes in format may lead to difficulties when reading data
written with the previous version or long time ago.
At the opposite, interchange activities are limited in time. Exchanged data in CEOS ICF are ephemeral
and has not to be kept. Possible changes of CEOS ICF will therefore have a less critical impact.

3.1.8. Sensor Description for Earth Observing Dynamic
Our intention was to use the sensor definitions present in document “A Strawman Proposal for a
Standard Sensor Description Format of Earth Observing Dynamic Sensors” (M-4). Unfortunately, these
definitions cannot be retained for the following reasons:
     These sensor definitions do not match (or very few) the information and the formalism found within
      the archives (see R-1). Scope of interchange format is to deal with data entities actually present in
      source and target formats, and is not to introduce new information.
     Sensor definitions of M-4 proposal are aimed to provide the necessary (and homogeneous)
      information that would enable further geolocation of the acquired data. Importance of this issue is to
      be considered when designing a new (or universal) archive format.

3.1.9. Open Archival Information System
In the same way as previous section, a possible contribution of principles and statements laid down in
“Reference Model for an Open Archival Information System (OAIS)”(M-5) has been assessed. An
interesting concept is defined in section “2.2.2. Information Package Definition” that introduces the
Preservation Description Information (see here below).
Preservation Description Information (PDI) is divided into four types of preserving information:
 Provenance that describes the source of Content Information (primary target of the preservation).
 Context describes how the Content Information (CI) relates to other information outside the
    Information Package (CI+PDI).
 Reference provides one or more identifiers, or systems of identifiers, by which the Content
    Information may be uniquely identified.
 Fixity that provides a wrapper, or protective shield, that protects the Content Information from
    undocumented alteration (for example use of check sum).
Some information may be found in section “5.1.3.4. Transformation” that introduces the concepts of
Reversible Transformation and Non-Reversible Transformation. This section concerns principally the
transcription activity.
Interoperability between OAIS archives is examinated in section “6.1. Technical Levels of Interaction
Between OAIS Archives”. This section identifies the four following schemas of collaboration:
-independent archives, -cooperating archives, -federated archives, and –archives with shared functional
areas.
                              Development of a baseband data archive                    reference GAEL-ICF-TN-060.2

          GAEL                                    interchange format                        issue 2 revision 3
                                                                                             date 13/04/2000
      Consultant                                       ICF Concepts technical note
                                                                                            page 19 of 30




3.2. High level requirements
     A good format for interchanging baseband data archive shall satisfy the three major constraints:
     1.   Portability – The format shall be suitable for a wide variety of storage and exchange technologies.
          Format shall be simple and managed by standard softwares.
     2.   Exhaustiveness – All the elements shall be kept that will allow post-processing‟s until a reliable end-
          user product.
     3.   Opening – The format shall be open for further introduction of new mission/instrument until new
          acquisition technologies.
                                Development of a baseband data archive                      reference GAEL-ICF-TN-060.2

             GAEL                                   interchange format                          issue 2 revision 3
                                                                                                 date 13/04/2000
         Consultant                                       ICF Concepts technical note
                                                                                                 page 20 of 30




4.   CEOS ICF DETAILED REQUIREMENTS
4.1. Baseband Interchange Unit (BIU)
     Interchange will concern only one segment (also called sub-segment (F-2) or sub-interval (F-5)) of one
     satellite at time. A segment is a period of acquisition during that the physical characteristics of the
     platform stay invariant (same spectral mode, same pulse frequency, same viewing angle…). All these
     parameters will be stored with only one value within the attached metadata file. It is therefore not
     foreseen to describe a list of products (catalog) as metadata of the exchange.
     This interchange unit (sometimes called dataset or datatake) is called Baseband Interchange Unit
     (abbreviation BIU). Data of one BIU are stored in a collection of files (at least two: 1 metadata file + 1
     signal data file).
     When the export facility desires to export more than one BIU on the same physical support, they will be
     sequentially stored until the physical end of recording mark is encountered (EOT for tape, directory
     contents for disk files and communication closing for electronic transfer).



4.2. Baseband Interchange Medium

     4.2.1. Multi-media exchange
     CEOS ICF has been basically foreseen to describe data stored on disk files. Nevertheless, exchange of
     baseband data may use the following types of supports:
          Disk files – series of files under the same directory,
          Tapes (DLT, Exabyte, DAT…) – series of sequential files separated by EOF mark and terminated by
           the EOT mark,
          Electronic transfer – series of disk files transferred using an appropriate file transfer protocol.

     4.2.2. File naming
     All the files of a same baseband interchange unit shall have the same prefix. This prefix is compound
     from values of mandatory fields identifying at least the mission, instrument, station, and acquisition
     date/time start/stop for the segment.
     The suffix shall identify the type of data (signal or metadata).

     4.2.3. Use of tapes and data packaging
     One or more tape(s )may be used to exchange BIU. In this case, standard archiving tools (UNIX:tar or
     gzip, cpio, dd; VMS:BACKUP; Windows: zip…) will be agreed between export and import facilities.

     4.2.4. Independence from medium and limitations of operating system
     Interchange is very often an occasional interface involving two organizations that should agree about the
     type of medium and possible limitations due to their operating systems. To reduce risks of
     misunderstanding, CEOS ICF shall allow to adopt the lowest profile of performances.
     Disk representation – File size: Size of files on disk may not exceed 2 31 bytes (approximately 2
     Gigabytes) because many operating systems (or versions of operating systems) do not allow file sizes
     greater than this limit (for example, the standard UNIX “tar” command deals with files whose size is less
     than 2 GB). When a file is greater, it may be split in a series of 2 GB files whose names record the
     sequence for further concatenation.
                                 Development of a baseband data archive                  reference GAEL-ICF-TN-060.2

             GAEL                                    interchange format                      issue 2 revision 3
                                                                                              date 13/04/2000
         Consultant                                     ICF Concepts technical note
                                                                                             page 21 of 30




     4.2.5. Data segmentation
     An other way to reduce the size of data shall be given allowing to export only a part of the acquired
     segment. Metadata shall record the elements (time/frame start/stop) allowing to accurately locate the part
     of segment being exchanged.



4.3. BIU Organization

     4.3.1. File structure
     To maintain CEOS ICF as simple as possible, an entire BIU will basically contain
          one unique Metadata file, and
          one or more signal data files.
     Other files (for example a “Tape Volume Label File”) could accompagn the BIU in the medium but these
     files are not part of the ICF specification. In order to allow ICF readers to clearly identify the metadata
     file, its first line will always contain a statement identifying it-self.

     4.3.2. File order
     The metadata file will always stand in first position within BIU on sequential access media like tapes or
     electronic transfer.



4.4. Metadata Management

     4.4.1. Metadata Organization
     Metadata are stored within groups of information organized in a hierarchical way. This organization shall
     allow to add a new group (for example for a new mission or instrument) under a certain node of this
     hierarchy, without changing the rest of the format.

     4.4.2. Language for Metadata Description
     The language to be used for the description of metadata should fulfill the following technical
     requirements:
    a)    Strict use of ASCII.
    b) Hierarchical description – Language shall allow to describe groups compound of one or more
       (sub)groups and terminal leaves.
    c)    Base data types – Integer, float, double, boolean and string data types shall be available and numerical
          values can be managed with the largest number of significant bits allowed by the intrinsic
          representation.
    d) User‟s data type – Programmer may built-in its proper data type (for example a UTC date-time)
       naming it for further use. Description of these user‟s data type will be joined to the BIU. CEOS ICF
       shall also takes into account the possibility that certain information shall not be described by their
       content but optionally by the location where they can be found. It can refer to a document or a server
       which can be respectively accessed through a reference or an URL address. The possibility to transfer
       blocks of information may be sometimes required.
    e)    Signal data schema – Language shall allow to define the structure of the signal data record within the
          relative metadata file. In such a way, import translation software could dynamically read and process
          signal data records.
                             Development of a baseband data archive                      reference GAEL-ICF-TN-060.2

         GAEL                                    interchange format                          issue 2 revision 3
                                                                                              date 13/04/2000
     Consultant                                         ICF Concepts technical note
                                                                                             page 22 of 30




f)      Lists – Language shall allow to represent lists with a variable and unlimited number of items. The
        empty list can be declared or can be considered as a default value.
g) Default value – Language shall allow to set a default value when no definition has been found in
   metadata file.
h) Undefined value – Language shall allow to declare that no value is known for a data entity.
i)      Mandatory/optional – Language shall allow to state that value for data entity is mandatory or optional.
        The language parser should be able to warn user when a data entity considered as mandatory has not
        been found with a value.
j)      Value range or enumeration – Language shall allow to define a range or a list of allowed values.
k) Conditional description - Language should allow to consider description for a data entity only if one
   or more conditions are fulfilled getting the value of one or more other data entities. For example data
   entities of the LANDSAT group have not to be defined if the mission is ERS…


 In addition this language should fulfill the following operational requirements:
l)      Parser able to ingest the description shall be Off-the-Shelf or is available freeware on Internet.
m) Language shall be maintained and a version should emerge considered as a standard.


 All these requirements will be collated with regard to the description languages already available
 (DEDSL, ODL, PVL, XML…). The data entity dictionary entries contained in Appendix B will be fully
 codified and documented using the selected language to stand within the “CEOS ICF” document.

 4.4.3. DED organization
 DED is organized in a hierarchical way and includes the following main sections at the first level of this
 hierarchy.

     Entity title                 Short description
     ICF_VERSION                  Information found in this section shall identify the version of the dictionary
                                  to be used to parse this metadata file.
     INTERCHANGE                  Information included in this section identifies the physical volume (see
                                  4.3.1) and the BIU contents. These last elements will be used to generate
                                  the metadata and signal data file names.
     PROCESSING                   This section contains the history information allowing to maintain the
                                  traceability of data through the main activities: -RECEIVING,
                                  -PRE_PROCESSING (TBC), -ARCHIVING and –EXPORTING.
     PLATFORM                     All the information generated on-board the platform is included in this
                                  section.
                                      ACQUISITION and ON_BOARD_COUNTER allow to precisely set
                                       the viewing period and the conversion between both dating systems.
                                      SEGMENT includes the identifiers (orbit, track, segment start/stop and
                                       image line number) of the granule.
                                      ORBITAL_BULLETIN retains parameters used by NORAD TLE
                                       (Two-Line Elements) to propagate the orbit.
                                      EPHEMERIS and ATTITUD are the state vectors registered on-board
                                       and transmitted in telemetry.
                          Development of a baseband data archive                          reference GAEL-ICF-TN-060.2

        GAEL                                  interchange format                               issue 2 revision 3
                                                                                                date 13/04/2000
    Consultant                                      ICF Concepts technical note
                                                                                               page 23 of 30




SIGNAL_DATA                    This section describes the way signal data files (one or more) are organized
                               and, in particular, the design of the record field per field.
QUALITY                        All the information that will form a Quality bulletin is reported in this
                               section. This bulletin is divided in three sub-sections relative to the various
                               critical steps: -ACQUISITION, -TRANSMISSION, -PRE_PROCESSING.
                           The number of observed defects (dead detectors, out of range radiometry,
                           synchronization loss, bit errors, missing lines) is reported. In addition, for
                           some of these defects, a list of indices (or locations) for that a defect has
                           been observed may be reported.
The complete list of CEOS ICF data entities is reported in Appendix B.

4.4.4. Mission-dependent section
An attempt has been made to try to unify the various missions / instruments according to their technical
characteristics. Using a hierarchical approach, each node includes attributes that are relevant only to it-
self and to its sub-tree (see figure 4.4.4).


                                                  SENSOR




           SCANNER                PUSH_BROOM                     MATRIX                         SAR
      DETECTOR_NUMBER         DETECTOR_NUMBER         VERTICAL_DETECTOR_NUMBER      PULSE_REPETITION_FREQUENCY
      SCANNING_PERIOD         SAMPLING_PERIOD         HORIZONTAL_DETECTOR_NUMBER    …
      …                       …                       …




                                             TO BE COMPLETED


                                  Figure 4.4.4.: Sensor characteristics tree.
Unfortunately, it seems difficult and not always easy to understand to pursue in this way.
We suggest an alternative solution, in that all the attributes specific (i.e.; not already foreseen in the
CEOS ICF DED) to a certain mission / instrument would be stored under a SPECIFIC section (example
“SPECIFIC_SPOT” or “SPECIFIC_LANDSAT_ETM”...).
This specific section will have the following characteristics:
     These sections will be located at the first hierarchical level, i.e. at the same level as the standard
      sections listed in 4.4.3.
     This specific section will be defined with the same formalism as for the standard section.
     Name and contents of this section will be agreed in close collaboration with the agency operating this
      satellite. Organization in charge of CEOS ICF configuration control would design a first proposal of
      hierarchy deriving from the information found within the “Satellite to Ground Station” document(s).
                              Development of a baseband data archive                      reference GAEL-ICF-TN-060.2

            GAEL                                  interchange format                         issue 2 revision 3
                                                                                              date 13/04/2000
        Consultant                                     ICF Concepts technical note
                                                                                             page 24 of 30




4.5. Signal Data Management

    4.5.1. No redundancy
    CEOS ICF shall not introduce duplication of data neither redundancy allowing to check a particular value
    from values found in other fields.

    4.5.2. No encryption
    CEOS ICF shall not introduce encryption. If the input data is encrypted and if at least one of the
    destination archives expect no encrypted data in input, data shall be decrypted.

    4.5.3. CRC and Checksum
    When an error has been detected within the transmitted frame and if a correction mechanism is available,
    correction shall be applied..

    4.5.4. No compression
    Signal data shall not be compressed except in the following cases:
         Compression is defined inside the transmitted frame and would require sophisticated processing to
          decompress it.
         Compression is processed by a standard tape driver.
         File based lossless compression is applied by using standard, well agreed, SW tools to increase the
          transfer throughput, for example over the network.

    4.5.5. PN decoded
    Like for all the studied formats (see R-1), transmitted data shall be PN-decoded.

    4.5.6. Data is frame synchronized
    Loss of synchronization will lead to replace corrupted bytes (or lines) by 0 value.
    Bit slip/bit insertion shall be corrected. No “guard-bytes” (F-5) shall be kept within the signal records.

    4.5.7. Chronological order
    When data are time-reversed (case of playback from on-board recorders), bytes and bits will be reversed
    to restore a chronological acquisition.

    4.5.8. Byte alignment
    Bytes are aligned on major and minor (case of Landsat) frames.
    CEOS ICF supports both (TBC):
         the native representation as found within the telemetry, and
         byte alignment per pixel, i.e. when number of bits is extended to match a multiple of 8 (example of
          ERS SAR data for that the 5-bits used in telemetry are increased to 8 bits).
    A flag will be present in metadata file to indicate if the byte-alignment per pixel has been processed or
    not. When this alignment has been processed, the original number of bits within the native representation
    will be recorded.
                             Development of a baseband data archive                     reference GAEL-ICF-TN-060.2

         GAEL                                    interchange format                         issue 2 revision 3
                                                                                            date 13/04/2000
     Consultant                                        ICF Concepts technical note
                                                                                            page 25 of 30




    4.5.9. Signal data record
    In order to allow an unique description of all the records containing signal data, length of records shall
    match the smallest repeated pattern within the transmitted telemetry.
    For example, the fact that the first transmitted frame of ERS OGRC HR format (figure 4.4.2.3.-1 of F-7)
    differs from the 28 following ones would oblige to make coincide one CEOS ICF signal data record with
    29 ERS frames (record length = 7424 bytes = 29 x 256 bytes).
    In this way, it would be possible to describe the signal data record (see “RECORD_DESCRIPTION” tag)
    from fields identifying the –synchronization code, -IDHT-ID, -IDHT header, -auxiliary data, and –
    measurement data.



4.6. CEOS ICF Version Management
    To maintain its durability, CEOS ICF shall be managed by an institutional Configuration Control Board
    (CCB). Indeed, the increasing number of missions together with the development of new techniques make
    necessary to envisage upgrades of this format.
    Scope of this section is to state some first principles for the management of new releases.

    4.6.1. Minor/major versions
    CEOS ICF organization is principally based on the metadata description. This choice provide a light way
    to introduce new mission / instrument just adding new group(s) and data entities. Such change shall lead
    to increase the minor version number of CEOS ICF.
    All the other changes (deletion of group(s) or data entities, change in hierarchy, change of the type of data
    entity, change of the mandatory/optional status of a data entity…) shall lead to increase the major version
    number of CEOS ICF.
    In its primary release, after acceptance of “CEOS ICF” document, the first version v1.1 will allow to
    process at least Landsat ETM+, SPOT 4 P/Xi/XS and ERS SAR missions/instruments.

    4.6.2. DED and software availability
    All the versions of the Data Entity Dictionary and possibly the software (tools, API‟s) allowing to parse,
    analyze and generate the metadata file will be made available on a site of CEOS WGISS (TBC).
    CEOS ICF version number (minor and major) shall be recorded as first data entity within metadata file.
                             Development of a baseband data archive                  reference GAEL-ICF-TN-060.2

          GAEL                                   interchange format                     issue 2 revision 3
                                                                                         date 13/04/2000
      Consultant                                      ICF Concepts technical note
                                                                                        page 26 of 30




APPENDIX A – BASEBAND DATA CONCEPT COMPLIANCE MATRIX
A.1. Introduction
     One of the highest user‟s requirement of applicable document A1 is “R-9) The ICF format shall be based
     on the CEOS Baseband Data concept.”.
     Scope of this section is to guarantee that requirements laid down in this document are fully compliant
     with Baseband Data Concepts (A2).



A.2. Compliance Matrix

            Baseband Data Concepts (A2) requirements                       CEOS ICF requirements
     High level requirements
     BDC.1      Baseband data should be able of reconstructing the
                original sensor output data stream.
     Signal data management
     BDC.2      Baseband Data should remove encoding techniques        4.5.3.
                used for transmission (PN, CRC).
                                                                       4.5.5.
                But Synchronization codes and zero fill can be kept.
     BDC.3      Telemetry stream should be frame synchronized and      4.5.6.
                byte aligned.
                                                                       4.5.8.
     BDC.4      De-multiplexing of the original serial stream that
                separates the video data from all the rest is
                mandatory.
     BDC.5      Bit/byte reversal of OBR data is to be considered      4.5.7.
                mandatory.
     BDC.6      Removal of duplicate data is mandatory.                4.5.1.
     BDC.7      Removal of bad data is mandatory.                      4.5.3.
     BDC.8      Ordering of data in time forward order is mandatory.   4.5.7.
     BDC.9      Proper time tagging of the stored data is mandatory.   4.4.3.       PLATFORM section
     BDC.10 All data gaps should be detected and occurrences           4.4.3.       QUALITY section
            should be recorded in the metadata.
     Metadata management
     BDC.11 The metadata will include information on the satellite, 4.4.3.          INTERCHANGE,
            sensor and mission, identification of facilities and                    PROCESSING and
            processors involved in the formation of product.                        PLATFORM sections
     BDC.12 The metadata should also include times of various          4.4.3.       PROCESSING section
            events such as acquisition time, down link time and
            transcription time.
     BDC.13 Other mandatory information: -special configuration,       4.4.4.       SPECIFIC section
            -mode of operation of sensor (on-board gain mode),
            -use of OBR, -use of relay station.
                      Development of a baseband data archive                 reference GAEL-ICF-TN-060.2

    GAEL                                  interchange format                     issue 2 revision 3
                                                                                 date 13/04/2000
Consultant                                    ICF Concepts technical note
                                                                                 page 27 of 30




BDC.14 Metadata will include methods for the identification of 4.4.3.       PLATEFORM.
       the information granules (ex. WRS).                                  SEGMENT section
BDC.15 Baseband data information granule should be clearly 4.4.3.           PLATEFORM.
       identified and time-tagged (on-board clock or another                ACQUISITION section
       secure time source).
BDC.16 PCD and on-board calibration information should be       4.4.3.      PLATFORM.
       included in the mandatory metadata for Baseband                      EPHEMERIS and
       Data.                                                                ATTITUD sections
                                                                            Calibration (TBD)
BDC.17 In the case the Baseband Data are time-tagged with       4.4.3.      PLATEFORM.
       the on-board clock, a method for converting it into                  ACQUISITION and
       universal time should be made available.                             ON_BOARD_COUNT
                                                                            ER sections
BDC.18 It should be possible to associate the baseband Data                 Times of signal data and
       granules with the (possibly) latest set of ephemeris                 ephemeris shall be
       elements, satellite attitude data and sensor pointing                within the same
       data.                                                                referential.
BDC.19 History of Baseband Data transcription can be            4.4.3.      PLATFORM.
       optionally recorded. History should include –data                    PROCESSING section.
       source, -version of processing/capture system, -facility
       identification, -format version, -data transmission
       quality, -data gap information.
                                Development of a baseband data archive                   reference GAEL-ICF-TN-060.2

              GAEL                                  interchange format                       issue 2 revision 3
                                                                                              date 13/04/2000
          Consultant                                     ICF Concepts technical note
                                                                                             page 28 of 30




APPENDIX B – CEOS ICF DATA ENTITY DICTIONARY


B1.   Introduction


      Name of entities will respect the following requirements:
           Language to be used is the US American.
           All the letters of the name will only use upper-case characters [A,Z] and the underscore character (_)
            to separate words.
           Use of abbreviations shall be minimized (example “NUMBER” in place of “NR”).
           Use of plural with letter “S” shall be minimized (example “ATTITUDE_NUMBER” in place of
            “NUMBER_OF_ATTITUDES”).



B.2. Hierarchical representation

      CEOS_ICF
              ICF_VERSION
                    MINOR
                    MAJOR
              INTERCHANGE
                    PHYSICAL_ORGANIZATION
                          VOLUME_NUMBER
                          VOLUME_INDEX
                          VOLUME_NAME
                    SATELLITE_ID
                          FAMILY_NAME
                          NUMBER
                          CODE
                    SENSOR_ID
                          FAMILY_NAME
                          NUMBER
                          CODE
                    ACQUISITION
                          START_DATE
                          START_TIME
                          STOP_TIME
              PROCESSING
                   RECEIVING
                         DATE
                                 START
                                 STOP
                          FACILITY
                                 COUNTRY
                                 AGENCY
                                 STATION
                                 FACILITY
                                 SOFTWARE_RELEASE
                          FORMAT_SYNCHRONIZER_DECOMMUTATOR                                                 Optional
                                 NAME
                                 RELEASE
                     PREPROCESSING                                                                         Optional

                            SOFTWARE
                                 NAME
                                 RELEASE                 ARCHIVING
                   Development of a baseband data archive               reference GAEL-ICF-TN-060.2

    GAEL                               interchange format                  issue 2 revision 3
                                                                            date 13/04/2000
Consultant                                ICF Concepts technical note
                                                                           page 29 of 30




                DATE
                       START
                       STOP
          EXPORTING
               DATE
                       START
                       STOP
                FACILITY
                       COUNTRY
                       AGENCY
                       STATION
                       FACILITY
                       SOFTWARE_RELEASE
                FILE_NAME_RADIX
    PLATFORM
         ACQUISITION
               TIME_START
               TIME_STOP
         ON_BOARD_COUNTER
               COUNTER_START
               COUNTER_STOP
         SEGMENT
               ORBIT_NUMBER
               TRACK_NUMBER
               SEGMENT_START     (time)
               SEGMENT_STOP      (time)
         ORBITAL_BULLETIN
               PRODUCED_BY
               UTC_DATE
               ORBITAL_INCLINATION
               RIGH_ASCENSION_NODE
               ECCENTRICITY
               ARGUMENT_OF_PERIGEE
               MEAN_ANOMALY
               MEAN_MOTION
               DRAG_COEFFICIENTS
                     NUMBER
                     LIST
         EPHEMERIS
               PRODUCED_BY
               NUMBER
               LIST
                     TIME
                     POSITION
                           X
                           Y
                           Z
                     VELOCITY
                           X
                           Y
                           Z
         ATTITUDE
               ANGLE
                     NUMBER
                     LIST
                           TIME
                           YAW
                           ROLL
                           PITCH
               VELOCITY
                     NUMBER
                     LIST
                           TIME
                           YAW
                           ROLL
                           PITCH
    SIGNAL_DATA
          NUMBER
          LIST
                  Development of a baseband data archive           reference GAEL-ICF-TN-060.2

    GAEL                              interchange format              issue 2 revision 3
                                                                       date 13/04/2000
Consultant                           ICF Concepts technical note
                                                                      page 30 of 30




               ORGANIZATION
                     DATA_SIZE
                     FILE_NUMBER
                     RECORD_LENGTH
                     FIRST_FILE_RECORD_LENGTH
                     FILE_NAME_RADIX
               BYTE_ORDER        (LSBF,MSBF)
               BYTE_ALIGNMENT
                     IS_PROCESSED_FLAG
                     BITS_PER_PIXEL
                           NATIVE
                           CURRENT
               DATA
                     INTERLEAVING_INDICATOR
                     BITS_PER_SAMPLE
                     SAMPLES_PER_BLOCK
                     CHANNELS_PER_BLOCK
                     BLOCKS_PER_RECORD
               RECORD_DESCRIPTION
                     FIELD
                           NUMBER
                           LIST
                                 NAME
                                 DEFINITION
                                 BYTE_START
                                 BYTE_STOP
                                 OCCURRENCE
                                 CODING_TYPE      (BINARY,ASCII)
                                 DATA_TYPE        (INTEGER,FLOAT,DOUBLE,STRING)
    QUALITY
          TRANSMISSION
               SYNCRONIZATION                                                       Optional
                     PATTERN
                     LOSS
                          NUMBER
               BIT
                     ERROR_NUMBER
                     ERROR_RATE
               FRAME
                     PROCESSED_NUMBER
                     ERROR_NUMBER
    SPECIFIC_LANDSAT7_ETM
          …
    SPECIFIC_SPOT1234
          …
         QUALITY
               ACQUISITION
                     DEAD_DETECTOR_NUMBER
                     OUT_OF_RANGE_RADIOMETRY
         PREPROCESSING
               MISSING_LINE
                     NUMBER
                     REPORTED
                            NUMBER
                            LIST

         …
    SPECIFIC_ERS_SAR_OGRC
          …

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:7
posted:4/18/2011
language:Irish
pages:30