Slide presentation Template - Download as PowerPoint by moneu


									WP9 Earth Observation Applications

                           DataGrid France 12 Feb. 2003 – WP9 – n° 1
          Global Ozone (GOME) Data Processing
                               and Validation
 The DataGrid testbed provides a collaborative processing
  environment for 3 geographically distributed EO sites (Holland,
  France, Italy)
 Level-1 data (raw satellite measurements) are analysed to retrieve
  the physical quantities : Ozone Profiles (Level-2 data)
 Ozone Profiles contain measurements of OZONE within a vertical
  column of atmosphere at a given lat/lon location above the Earth’s
 Two different Grid processing techniques are being investigated
    OPERA (Holland) - Tightly coupled - based on MPI
    NOPREGO (Italy) - Loosely coupled - based on Neural Networks
 Selected Profiles are validated by comparing them with ground-based
  LIDAR measurements in a coincident area and time

                                                     DataGrid France 12 Feb. 2003 – WP9 – n° 2
                                  EDG EO challenge: Processing /
                                  validation of 1y of GOME data
          Raw satellite data
      from the GOME instrument
      (~75 GB - ~5000 orbits/y)

                           Level 1
                                     (example of 1 day total O3)                           LIDAR
                                                                                      data (7 stations,
Processing of raw GOME
                                                                                      2.5MB per month)
 data to ozone profiles.
2 alternative algorithms
   ~28000 profiles/day
                                       Validate some of the
                                  GOME ozone profiles (~106/y)
                                   Coincident in space and time
                                  Ground-Based measurements

   Level 2                 DataGrid
                         environment                               Visualization
                                                                     & Analyze

                                                                    DataGrid France 12 Feb. 2003 – WP9 – n° 3
                    EO Use Case File Numbers
               1 Year of GOME data :
  Dataset          Number of files                        Size
Level 1                    4,724                          15 Mb
Level 2
                         9,448,000                         10 kb
Level 2
                         9,448,000                         12 kb
Opera (KNMI)
                             12                         2.5 Mb
Lidar (IPSL)
    Total:          18,900,736 files                   267 Gb
             (just one part of a 5-year dataset)
                                          DataGrid France 12 Feb. 2003 – WP9 – n° 4
               GOME Ozone Profile Processing

Select geographic area and time to be
Retrieve list of available Level-1 data
 files from the Product Catalogue
Transfer files to the GRID
Perform Level-1 Processing, retrieve
 Ozone Profiles
Visualize results

                                DataGrid France 12 Feb. 2003 – WP9 – n° 5
                                 GOME Ozone Profile Validation

                             ERS/GOME satellite

    50 km
                                  OZONE LAYER
    10 km

 Goals of the DataGrid application
   to point out : the capability to validate satellite data
   with all ground based data available in an easy way:
    Comparison of ozone profiles provided by satellite
   with lidar data in differents locations and times (see
   the web portal)
    Statistical comparison and analysis in order to
   improve algorithms.                                          Lidar at the Haute
                                                               Provence Observatory
                                                              DataGrid France 12 Feb. 2003(5/02/2003)
                                                                                           – WP9 – n° 6
                                         Validation Processing Sequence

            Satellite data validation                            Lidar site

                     Level 2 Catalogue
 Level 2 Catalogue

Queries and data information retrieval
from the Gome Level 2 orbit or pixel
metadata catalogues
                                                                      Queries and data information
                                           3                          retrieval from the Lidar metadata
                     Submission of the         GRID Portal
                      Job in the GRID                                 catalogue

                                                                                                Lidar data
                                 GRID                                   1                       catalogue
           Storage Elements
            with Lidar data
                                                4            When completed
                                                             comparison between
                                                             lidar and satellite
                                                             ozone profiles
Storage Elements with
    Gome L2 data

                                                                                   DataGrid France 12 Feb. 2003 – WP9 – n° 7
DataGrid France 12 Feb. 2003 – WP9 – n° 8
DataGrid France 12 Feb. 2003 – WP9 – n° 9
                                                                           Processing Sequence
                                                   1. Search Level-1 catalogue
                 Web Portal                                                                     EO Product
                                                 12. Return new Level-2 products
2. Retrieve Level-2 products

    3. Level-2 Products already
    registered in RC?                                EO                                     EO Product
    Yes? 4. Return available
    Level-2 products
                                                 Grid Engine                                 Archive
    No? 5. Perform GRID
    processing on-the-fly       
                                                                                   6. Transfer Level-1
                                               User Interface                      data from Archive to
   EO Replica                                                                      the Grid
    Catalogue               7. Register Level-1 data

    11. Register level-2 data                        EDG
    8. Submit jobs to process Level-1 data         Resource
         EDG                                        Broker                                       EDG
      Computing                          9. Process Level-1 data                               Storage
       Element                                                                                 Element
                                             10. Transfer Level-2 data to SE

                                                                                          DataGrid France 12 Feb. 2003 – WP9 – n° 10
                                       The EO WebMap Portal
 The scientist interacts with the GOME information system using the
  WebMap portal

 This allows the user to choose a geographic area and time interval to be

 The WebMap portal retrieves and displays the available GOME products on
  the map

 It does this by communicating with several background services :

    Web Map Server – provides map images

    Web Service Broker – provides GRID web services

    EO Grid Engine – integrates the Data Grid services

Products which were recently requested will be already available
Other products will be processed “on     the fly” using the GRID resources
                                                       DataGrid France 12 Feb. 2003 – WP9 – n° 11
The Web Portal Integrates several components of
 the EO Infrastructure
The EO GRID Engine provides a standard way for
 Web Clients to request GRID operations
The Web Portal and the EO Grid Engine allow the
 complexity of the underlying components to be
 hidden from the user
This Framework can be adapted to provide new
 tools for the satellite end-users

                                    DataGrid France 12 Feb. 2003 – WP9 – n° 12

To top