3. Guidelines for Opendap Installation _ Configuration by yaoyufang

VIEWS: 0 PAGES: 40

									                           Research Project co-funded by the
                                European Commission
                            Research Directorate-General
                                th
                               6 Framework Programme
                               FP6-2002-Space-1-GMES
                            Ocean and Marine Applications
                           Contract No. AIP3-CT-2003-502885




                                    MERSEA IP
                        Marine EnviRonment and Security for
                       the European Area - Integrated Project




              Guidelines for Data Provider
            - Quick Opendap Installation &
                Configuration Guide -



Ref:   [MERSEA-WP06-CLS-UMAN-002_1C_opendap_installation.doc]
       November 2005




Author: CLS




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                  1
                                                                         Guidelines for Data Provider
                                                     Quick OpenDAP Installation & Configuration Guide


                                         Table of Content

1.     INTRODUCTION                                                                                 4

1.1    Object of the Document                                                                       4

1.2    About Mersea and its Dissemination Services                                                  4


2.     DATA TRANSPORT AND EXCHANGE PROCEDURES                                                       5

2.1    Mersea Recommendations                                                                       5

2.2    What is OpenDAP                                                                              6
  2.2.1   OpenDAP Overview or the 10 Reasons to Use OpenDAP                                         6
  2.2.2   OpenDAP Servers                                                                          10
  2.2.3   OpenDAP Clients                                                                          11
  2.2.4   How to use OpenDAP                                                                       13
  2.2.5   FTP versus OpenDAP                                                                       14
  2.2.6   More on OpenDAP                                                                          15

2.3    Our Experience (AS versus GDS)                                                              16


3.     GUIDELINES FOR OPENDAP INSTALLATION & CONFIGURATION                                        20

3.1    Pre-Requesites (Technical Requirements)                                                     20
  3.1.1    Hardware                                                                                20
  3.1.2    Operating System                                                                        20
  3.1.3    Software                                                                                20
  3.1.4    Software Versions and Environment                                                       21

3.2    Aggregation Server (AS)                                                                     22
  3.2.1    Introduction                                                                            22
  3.2.2    Documentation & Contacts                                                                22
  3.2.3    Download & Installation                                                                 22
  3.2.4    Configuration Information                                                               23

3.3    GrADS Data Server (GDS)                                                                     25
  3.3.1   Introduction                                                                             25
  3.3.2   Documentation & Contacts                                                                 25
  3.3.3   Download & Installation                                                                  26
  3.3.4   Configuration Information                                                                27


4     ROUTINE OPERATIONS WRT MERSEA INTEGRATED SYSTEM                                             28

4.1    Monitoring Function                                                                         28
  4.1.1   Introduction                                                                             28
  4.1.2   Routine Opendap Update                                                                   28
  4.1.3   Regularly Check Web Server Availability                                                  28
  4.1.4   An Application: the OpenDAP Data Status                                                  29

4.2    Synchronisation Process                                                                     29
  4.2.1    Introduction                                                                            29
  4.2.2    Synchronisation Process                                                                 29
  4.2.3    An application: the Federating Live Access Server                                       32



Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                      1
                                                                 Guidelines for Data Provider
                                             Quick OpenDAP Installation & Configuration Guide
4.3    The Context File                                                                     33
  4.3.1    Introduction                                                                     33
  4.3.2    Context File Format                                                              33


APPENDIX A: SCRIPT TO STOP/START TOMCAT                                                    35

APPENDIX B: GDS MAIN CONFIGURATION FILE                                                    37

APPENDIX C: A PERL PROGRAM
TO CHECK REGULARLY WEB SERVER AVAILABILITY                                                 38

APPENDIX D: PARAMETER FILE FOR “VERIFY_WEB.PL”                                             39




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                              2
                                                                     Guidelines for Data Provider
                                                 Quick OpenDAP Installation & Configuration Guide




                           Document Change Record
        Author                           Modification                      Issue         Date
F. Blanc,               Creation                                             1A      20 july 2005
T. Jolibois,
JM. Epitalon
G. Manzella             Wrt Mersea Recommendations                          1B       september
T. Jolibois             Change in software file name and download place                2005
T. Jolibois             Change in download place of thredds.war             1C     november 2005
                        3.2.4 Delete two deprecated links




    APPLICABLE DOCUMENTS / REFERENCE DOCUMENTS

RD 1      MERSEA-WP06-CLS-UMAN-001-A
          Guidelines for Mersea Data Provider:
          Gridded Data Product Format

RD 2      MERSEA-WP06-CLS-UMAN-002-A
          Guidelines for Mersea Data Provider:
          Quick Opendap Installation & Configuration Guide

RD 3      MERSEA-WP06-CLS-UMAN-003-A
          Model Document:
          Catalogue for Mersea Data Provider

RD 4      MERSEA-WP06-CLS-UMAN-004-A
          Model Document:
          User Manual for Mersea Ocean Data Products

RD 5      MERSEA-WP06-CLS-UMAN-005-A
          Quick Opendap User Guide




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                      3
                                                                       Guidelines for Data Provider
                                                   Quick OpenDAP Installation & Configuration Guide



1. INTRODUCTION


1.1     Object of the Document
This document gives guidelines to Mersea data provider on which OpenDAP to select, how to find easily
the information as well as our feedback on the Grads and Aggregation Data Server (GDS and AS) for
installation, configuration & Mersea federating operations.

Besides this introduction, the document contains 3 sections:
1. Mersea recommendations for the OpenDAP technology for data dissemination together with an
   OpenDAP overview, the list of current OpenDAP servers and clients and pro’s and con’s between
   GRADS and AS.
2. Guidelines for OpenDAP installation & configuration including information on where to find it, which
   version to use, where to get the installation & configuration documentation, and our feedback for AS
   and GrADS prerequisites, installation and configuration.
3. Routine operations needed for individual OpenDAPs and Mersea Federation System to reach an
   operational Mersea Integrated System (e.g. monitoring function and synchronization with a front-end
   server like Live Access Server…).

It is in a draft version and we still are in an iteration process to produce a good reference document.
Elements written in red are to be checked or written. For any correction or enhancements, please
iterate with webmaster-mersea@mersea.eu.org and help us improve this reference document.

The current release of the document is available at
http://w3.mersea.eu.org/html/remote_sensing/framework.html#distribution


1.2 About Mersea and its Dissemination Services
MERSEA stands for Marine Environment and Security for the European Area. The MERSEA ocean
monitoring system is envisioned as an operational network that systematically acquires data (earth
observation from satellites, in situ from ocean observing networks, and surface forcing fields from
numerical weather prediction agencies). These data from diverse sources are combined and merged with
numerical ocean circulation models (i.e. assimilated) to produce best estimates of the actual state of the
ocean (nowcasts) and forecasts. That information is then disseminated to enable various users to
develop specific applications. The project itself has identified areas of Special Focus Experiments on
seasonal weather forecasting, and on ecosystem modelling in regional and shelf seas. Specific
applications on marine safety concern improved wave forecasts, offshore operations, ship routing, and oil
spill drift.

Mersea is serving and communicating with different thematic communities. So, the challenge for Mersea
is to find a balance and to merge the practices and experiences of these different thematic communities
into a coherent set of ocean products that will take into account specific features of the products and
provide simultaneously interoperability to these surrounding communities and networks.

A common unified framework for product coherency and standardisation as well as data transport and
exchange procedures has been set to provide a coherent set of dissemination services with the ability to
access the data in an interoperable manner from client applications, relying on a decentralised but
compatible system architecture for distribution on Internet,

More information on Mersea can be found on the MERSEA website: http://www.mersea.eu.org/.




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                          4
                                                                       Guidelines for Data Provider
                                                   Quick OpenDAP Installation & Configuration Guide



2. DATA TRANSPORT AND EXCHANGE PROCEDURES


2.1       Mersea Recommendations

Mersea first requirements and recommendations are issued from the current dissemination services of
existing data centres and their own requirements and revision process as well as results from the
MERSEA-Strand-1 intercomparison framework. They requirements can be resumed as follows:

   To distribute / transport data via Internet (push/pull integration)
    (with minimum constraint)

   To manage large data sets, numerous data flows and numerous users

   To allow synctatic and semantic operability
    (consistent format representation across data sets,
    consistent semantic representations of the data)

   To preservate data and products

   To rely on open source technology
    associated to reliable, sustainable, efficient operations

Mersea main recommendation was to let the data be managed independently at participating sites and to
rely consequently on a decentralised but compatible system architecture for distribution on Internet.
MERSEA vision is that of a single data access protocol for the exchange of data between network based
applications.

To allow syntactic and semantic operability, MERSEA selected the Opendap technology as a good
vector, at least for gridded product dissemination. Opendap represent, in the MERSEA vision, a layer
above TCP/IP providing a consistency (syntactic and semantic) not available in existing protocols, such
as FTP. FTP can of course remain an excellent complement to OpenDAP technology. Data centres are
also recommended to keep this offer to users.


Notice:

1- Current choice is also in agreement with the IOOS Data Management And Communication plan
(DMAC, http://dmac.ocean.us) and the Godae implementation plan (http://www.bom.gov.au/GODAE/ ).,
and

2- Choosing OpenDAP does not mean that OpenDAP is a static part of software or protocol. In fact we
believe that all elements of the system, regardless of their current designation, will evolve in time.




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                       5
                                                                           Guidelines for Data Provider
                                                       Quick OpenDAP Installation & Configuration Guide



2.2      What is OpenDAP

2.2.1 OpenDAP Overview or the 10 Reasons to Use OpenDAP

     "OpenDAP" stands for "Open-source Project for a Network Data Access Protocol".

      o OpenDAP is both the name of a non-profit organization and the name of the protocol which the
        OpenDAP organization has developed (see http://www.opendap.org).

      o OpenDAP is a Data Access Protocol (DAP1) for requesting and transporting data across the web.


     OpenDAP software is open-source (XML native + Java servlets) and is intended to facilitate
      building community driven projects for developing end-to-end systems which can locate, understand,
      access, and use scientific data. The design of the OpenDAP software was based on these two
      considerations:

    o    data are often most appropriately distributed by the individual or group that has developed them;

    o    and the user will in general like to access data from the application software with which they are
         most familiar.


   OpenDAP is widely used:

      o OpenDAP has been serving the marine community since 1995 and serves already a broad range
        of data, including oceanographic, atmospheric, and even astronomical data (cf. the Opendap
        master data sets list).

      o OpenDAP is an operational component of IOOS for access to gridded data (cf IOOS Data
        Management And Communication plan –DMAC–, http://dmac.ocean.us/).


     OpenDAP is a software framework that simplifies all aspects of scientific data networking,
      allowing simple access to remote data:

      o Local data can be made accessible to remote locations regardless of local storage format by
        using servers. OpenDAP converts transparently from a number of commonly used data formats
        (NetCDF, GRIB, HDF4, MATLAB®, ASCII) , into the format appropriate for the analysis package.

      o Existing, familiar data analysis and visualization applications can be transformed into clients (i.e.,
        applications able to access remote served data, like IDL, Matlab, Ferret, netCDF Operators,
        spreadsheets operators, LAS2, MapServer).



1
    "The Data Access Protocol--DAP 2.0" document has been submitted to NASA/ESE Standards Process Group
    (SPG) as a candidate draft standard (June 2004). It is part also of the DMAC strategy.
2
    LAS is an Opendap client-application which enables user-friendly visualisation and/or data extraction without
    specific knowledge of Opendap technology.
        LAS allows various ways to visualise the data (maps, sections, time evolution, etc.) and offers means for
         intercomparison between two kinds of comparable data (e.g. remote sensing data and model outputs) and
         for basic statistics (average and standard deviation).
        LAS enables Opendap federation and visualise various data sets from geo-dsitributed Opendap locations.
         (cf. http://las.mersea.eu.org)




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                                  6
                                                                            Guidelines for Data Provider
                                                        Quick OpenDAP Installation & Configuration Guide

   OpenDAP has been designed to distribute large data sets on Internet (uniform access to
    scientific data on the Internet - HTTP protocol -) and minimize the barriers to sharing them:

     o This highly distributed system allows users to control the distribution of their own data and the
       way they access data from remote sites.
                                                                                                       3
     o OpenDAP also allows a client to request only a subset of a dataset (and aggregation ) (selection
       of parameters at given space and time windows, compose a time series) as well as function
       evaluation like statistics functions (basic operation between parameters, mean computation, use
       of thresholds, etc.).

     o At the other end, an OpenDAP client, integrated into familiar analysis and visualization software
       packages access the data. Alternatively, a simple internet browser may be used as a client and
       allows to download data from the server.


   OpenDAP scheme:




             Note: OpenDAP was formerly called DODS (Distributed Oceanographic Data System).


   The Opendap technology allows the users, where ever he is, to access whatever data they
    require in a form they select (image, ASCII, NetCDF, etc.), using client applications they already
    are familiar with (IDL, Matlab, Ferret, netCDF Operators, spreadsheets operators, LAS, MapServer,
    etc.). Alternatively, a simple internet browser may be used as a client and allows to download data
    from the server.


   The Opendap technology provides strong support for data stored in NetCDF as well as for
    users of NetCDF enabled programs. Support for clients that utilize netCDF is particularly simple, as
    the netCDF native libraries have been OpenDAP-enabled so that a simple relinking of an application
    is often sufficient to OpenDAP-enable it.




3
    The Opendap Aggregation Server allows also to compose a time series of files and retrieve it in one access.



Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                                    7
                                                                                Guidelines for Data Provider
                                                            Quick OpenDAP Installation & Configuration Guide

        OpenDAP is not a data catalogue, neither a mechanism of transferring files as such, but a
         mechanism for transferring the information contained in files4 by using the DDS, the DAS and
         the DataDDS.

         o OpenDAP is associated with data stored as files 5.

         o The protocol may be thought of as XML-tagged data descriptions (metadata) with associated
           non-sparsed (binary data) content.

         o The data transport protocol relies on HTTP for requesting and providing data across the World
           Wide Web.

         o Within the data protocol the so-called "syntactic metadata", which describes the structure of the
           data (Grids, Sequences, Structures, etc.) is mandatory, however, most semantic information is
           optional.

              The DDS (data type object) express the
               structure of the numeric data, ie.
               synctatic metadata

              The DAS (attribute object) contain all
               relevant semantic metadata configured
               according to the structure expressed in
               the DDS.

              The DataDDS (data object) contain the
               numerical data itself in the linear form
               generated by a depth-first traversal of
               the structure expressed in the DDS.




     There are fours components to OpenDAP: servers, client libraries, a format-neutral data
      transport protocol and binaries generated. OpenDAP has already developed:

         o a flexible data model

         o a transport protocol (DAP2)
            DAP 2.0 uses HTTP to frame the requests and responses



4
        This requires that the server open and parse the datafile containing the information to be transmitted. And since
        the server will be processing the contents of the data file, OpenDAP was also designed to permit the client to
        instruct the server to return only data which fit the conditions in a query from client to server.
5
        Since files are stored locally, data can simultaneously be made accessible via FTP by enabling a FTP access on
        the local data storage. This is a system issue.



Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                                         8
                                                                     Guidelines for Data Provider
                                                 Quick OpenDAP Installation & Configuration Guide
    o software frameworks
       C++ and Java implementations of the data model and transport protocol
       a set of OpenDAP servers (see section 2.2.2).
       a set of OpenDAP clients (see section 2.2.3).

    o binaries generated
       Binaries are built for a certain numbers of platforms/operating systems, such as IRIX, Solaris,
         Linus, OSF




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                       9
                                                                       Guidelines for Data Provider
                                                   Quick OpenDAP Installation & Configuration Guide




2.2.2 OpenDAP Servers

The major strength of previous OpenDAP-enabled data servers has been in providing access to a
relatively small number of large gridded datasets or model results.

Because the expected data formats vary, there are different kinds of OpenDAP servers6; for example, a
JGOFS OpenDAP server furnishes its data in JGOFS format.

All Opendap servers are compatible within a client-application.

    Currently available OpenDAP servers:

OpenDAP netCDF Server                        This Opendap Data Server makes netCDF data available
OpenDAP HDF Server                           This Opendap Data Server makes HDF 4 data available
OpenDAP RDBS Server                          OpenDAP RDBS (Relational DataBase Server) makes
                                             RDBMS data that is data contained in a Relational
                                             Database system available. The DRDS uses JDBC to
                                             access the data in the RDBMS.
OpenDAP Test Server                          The test server serves pseudo-data (ie. made up by the
                                             server but consistent).
                                             Very useful when you want to test client software on a data
                                             structure for which you don't have an example.
 OpenDAP JGOFS Server                        This Opendap Data Server makes JGOFS data available
                       developed for JGOFS (cf. http://usjgofs.whoi.edu/).
 http://usjgofs.whoi.edu/datasys/jgdb_docs/j
gsys.html
OpenDAP FreeForm Server                      This Opendap Data Server makes data in an ‘home-grown
                                             format’ available .
                                             Freeform software package provides a flexible method for
                                             specifying data formats thus allowing the FreeForm server
                                             to serve a variety of data formats.
Aggregation Server (beta release)            The Aggregation Server (AS) allows to “aggregate”
                    developed by Unidata     multiple datasets into one dataset.
http://www.unidata.ucar.edu/projects/THRE    The server works with local netCDF file as well as data
DDS/tech/aggServer/AggServerStatus.html      accessible from a DAP-compliant server.
GDS                                        GDS (GrADS Data Server) services can be provided for
                        developed at COLA. any GrADS-readable dataset (GRIB, Binary, NetCDF, HDF,
http://grads.iges.org/grads/gds/index.html BUFR, and GrADS station data format).
                                           To be used with GrADS client-applications, the GDS unifies
                                           all these data formats into a NetCDF framework. The
                                           analysis capability allows users to retrieve the results of an
                                           operation applied to one or more datasets on the server.
                                           Examples of analysis operations include basic math
                                           functions, averages, smoothing, differencing, correlation,
                                           and regression. The GDS supports operations that can be
                                           expressed in a single GrADS expression.
Matlab Server                                This Opendap Data Server makes Matlab binary data
                                             available

6
    An OpenDAP server can be created by relinking an existing application, or by writing a new one. To be
    eligible for conversion to a OpenDAP server, an application must make use of one of the data access
    APIs that OpenDAP supports.



Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                         10
                                                                       Guidelines for Data Provider
                                                   Quick OpenDAP Installation & Configuration Guide
DSP Server                                    This Opendap Data Server makes DSP data available.
                                              DSP is a file format used by a University of Miami satellite
                                              package.
 Dapper Server                                The Dapper makes in-situ data (very large numbers of
          developed at NOAA/PMEL/EPIC.        individual files, profiles or time series) available via
 http://www.epic.noaa.gov/epic/software/dap   Opendap. Dapper also serves gridded datasets.
per/                                          Dapper is consistent with the OpenDAP-standard sequence
                                              structures utilized by the GrADS in situ Data Server. Data
                                              are stored using the EPIC netCDF convention.


   OpenDAP servers not yet publicly available:

OpenDAP HDF5 Server                           This Opendap Data Server makes HDF 5 data available.
UCAR/HAO CEDAR Server                         This Opendap Data Server makes CEDAR data available.
UCAR/HAO FITS Server                          This Opendap Data Server makes FITS data available.




2.2.3 OpenDAP Clients

There is no set appearance or functionality for a OpenDAP client; it can be implemented in a variety of
ways, and perform any functions that its users require.

The OpenDAP client uses OpenDAP functions to request data from the OpenDAP server, and to interpret
the results received from the server into a particular data format.

The data request functions use the http protocol, sending an enhanced URL to the server. The data
interpretation functions translate the data sent by the server into the data format expected by the rest of
the application.

   Currently available OpenDAP client software:

CDAT (Climate Data Analysis Tools)     a software infrastructure that forms an integrated
                    developed at PCMDI environment for solving model diagnosis problems.
OpenDAP geturl                                a simple command-line client for testing datasets.
                                              It is available as part of both the C++ and Java
                                              implementations.
Matlab Command-line client                    a Matlab command-line tool that reads OpenDAP data
                                              directly into Matlab variables.
                                              Note: Matlab (a commercial data analysis and visualization
                                              package) is required.
OpenDAP Matlab Toolkit (GUI)                  a GUI for accessing oceanographic data from DAP-
                                              compliant servers in Matlab.
Ferret                                   a data analysis and visualization package that can access
                available from NOAA/PMEL OpenDAP data.
GrADS                                      a data analysis and visualization package for earth system
                       available from COLA data.
IDV (Integrated Data Viewer)            a Java application for visualizing and analyzing geoscience
              developed at UCAR/Unidata data.
                                        The IDV was and uses the VisAD library for data and
                                        display models as well as other Java utility packages.
LAS (Live Access Server)              a highly configurable Web server designed to provide
             available from NOAA/TMAP flexible access to geo-referenced scientific data.
                                      LAS, by default, uses Ferret (see above) to produce plots



Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                               11
                                                                    Guidelines for Data Provider
                                                Quick OpenDAP Installation & Configuration Guide
                                          of datasets which means the LAS can be configured to
                                          display data from OpenDAP servers
ncBrowse                                  a Java application that provides flexible, interactive
                                          graphical displays of data and attributes from a wide range
                                          of netCDF data file conventions.
                                          Version 1.3.3Beta5 and later can access OpenDAP servers
ncdump                                  a utility for displaying netCDF files in the netCDF CDL
              developed at UCAR/Unidata (Common Data Language).
NCO (NetCDF Operators)                 NCO is a set of command line tools that perform operations
         an Open Source project hosted (e.g., average or concatenate) on netCDF or HDF files.
                   by SourceForge.net
ncview                                    a visual browser for netCDF gridded data.
OpenDAP Data Connector (ODC)              The ODC is a graphical application to allow to search for
                                          OpenDAP datasets (declared in the Opendap master list),
                                          download them to he user machine, save them, and import
                                          them into client applications like IDL, Matlab, SPSS, Excel,
                                          or into databases such as Access and Oracle.
                                          It also has viewing and plotting capabilities.
                                          Compatible with Unix/Linux, Windows, and Mac.



   Other applications that can access DAP-compliant servers:
Because OpenDAP servers are accessed through URLs, any application that can dereference a URL can
access those servers.
Generic Web Browser                       Thus, any generic web browser (e.g., Internet Explorer,
                                          Mozzilla, or Netscape) can access those data in a restricted
                                          manner.
                                          Since generic web browsers do not understand the
                                          OpenDAP protocol, the user will need some understanding
                                          of the protocol.
                                          A brief explanation of how to access OpenDAP data in a
                                          web browser is available at the OpenDAP FAQ 'Using a
                                          Web Browser with OpenDAP servers'.
Spreadsheet Applications
                                          Since most spreadsheet applications (e.g., MS Excel and
(e.g., Excel, StarOffice)
                                          StarOffice) can open URLs, they can access those data in
                                          a restricted manner.
                                          However, since spreadsheet applications do not
                                          understand the OpenDAP protocol, the user will need some
                                          understanding of the protocol.
                                          A brief explanation of how to access OpenDAP data from a
                                          spreadsheet is available at the OpenDAP FAQ Using a
                                          Spreadsheet Application with OpenDAP servers.




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                       12
                                                                 Guidelines for Data Provider
                                             Quick OpenDAP Installation & Configuration Guide


2.2.4 How to use OpenDAP

See RD5.




           OpenDAP Web Interface                 OpenDAP Data Request Form on the Web

                                                   Cf. http://opendap.aviso.oceanobs.com/
                                                                thredds/dodsC/
                                                        merged/h/msla_oer_merged_h
                                                                 ?time[0:1:195],
                                                 NbLatitudes[0:1:914],NbLongitudes[0:1:1079],
                                                         Grid_0001[0:1:0][0:1:0][0:1:0]




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                  13
                                                                      Guidelines for Data Provider
                                                  Quick OpenDAP Installation & Configuration Guide


2.2.5 FTP versus OpenDAP

Example with CNES/CLS/AVISO/DUYACS altimetry data

                               FTP protocol                            OpenDAP protocol
Data download ftp://ftp.cls.fr/pub/oceano/AVISO/NRT-        http://opendap.aviso.oceanobs.com/thred
              SLA/maps/oer/merged/h/msla_oer_merg           ds/dodsC/merged/h/msla_oer_merged_h
              ed_err_18861.nc.gz                            ?time[0:1:185],NbLatitudes[0:1:914],NbLo
              (1,47 mo, global file, all parameters)        ngitudes[0:1:1079],Grid_0001[0:1:0][0:1:0
                                                            ][0:1:0]
Working steps 1. The data must be imported / down-          1. Define first your datasets
                 loaded.                                         environment (temporal window,
              2. The data may have to be converted               sampling characteristics) thanks to
                 into a format that can be read by the           constraint expression in the URL
                 data analysis/processing program.               address
              3. Subsetting will be done at this stage.     2. Then download only the data you
                                                                 requested.
                                                            The download concern selected data plus
                                                            needed conversion if required to the
                                                            client format (binary/ascii).
Comment         •    FTP protocol                           •    HTTP protocol
                     through a pathname reference to             The user specifies a URL, which may
                     some local disk file                        point to either a local or a remote
                                                                 dataset.
                     No subsetting capabilities, all the   •    DAP added value
                      dataset is down-loaded                     Breadth of formats,
                                                                 Possibilty of subsetting, ie. select
                                                                 the data/ variable of interest plus
                                                                 geographical extraction,
                                                                 Possibility of aggregation, ie.
                                                                 allows a large collection of data sets
                                                                 to appear as a single TZYX virtual
                                                                 dataset,
                                                                 Push-pull integration
                                                                 Many application clients
                                                                 (nctools, matlab, Ferret, LAS, etc.)

                                                                 Show the metadata environment
                                                                  (data discovery, data search)
                                                                 If the data sets semantics are
                                                                  known, the data may be subsetted
                                                                  based on these semantics.
                                                                 Sampling and searching is
                                                                  performed on the distribution server
                                                                  machine. This reduces Internet
                                                                  traffic as well as decreasing the
                                                                  load on the local machine
Log tracks      Possibility to know                         Possibility to know
                • The country origin of the visitors        • The country origin of the visitors
                • Number of file requested per month        • Number of file requested per month
                • Volume acquired per month                 • Volume acquired per month
                • Unique IP address per month               • Unique IP address per month
                   (and who comes back)                        (and who comes back)
                                                            • Format and subsetting parameters




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                            14
                                                                  Guidelines for Data Provider
                                              Quick OpenDAP Installation & Configuration Guide


2.2.6 More on OpenDAP

All on OpenDAP                           http://www.opendap.org/
(formerly known as DODS)
OpenDAP servers                          http://www.opendap.org/faq/whatServers.html
o Aggregation Server (AS) (Unidata)      http://www.opendap.org/faq/whatServers.html#agg
o GrADS Data Server (GDS) (Cola)         http://www.opendap.org/faq/whatServers.html#gds
OpenDAP clients                          http://ww.opendap.org/faq/whatClients.html
o NCBrowse                               http://www.opendap.org/faq/whatClients.html#ncbrowse
o NCO (NetCDF Operators)                 http://www.opendap.org/faq/whatClients.html#nco
o NCview                                 http://www.opendap.org/faq/whatClients.html#ncview
o Matlab Command Line                    http://www.opendap.org/faq/whatClients.html#ml-cmdln
o Matlab GUI                             http://www.opendap.org/download/matlab_gui.html
o Integrated Data Viewer                 http://www.opendap.org/faq/whatClients.html#idv
o Climate Data Analysis Tools            http://www.opendap.org/faq/whatClients.html#cdat
o Ferret                                 http://www.opendap.org/faq/whatClients.html#ferret
o LAS (Live Access Server)               http://www.opendap.org/faq/whatClients.html#las
o OpenDAP Data Connector                 http://www.opendap.org/faq/whatClients.html#odc
o Spreadsheet Applications (Excel)       http://www.opendap.org/faq/whatClients.html#spreadsheet
Master list of data sets
                                         http://www.opendap.org/data/
served with OpenDAP


Getting started with OpenDAP          http://www.opendap.org/getStarted/
o What will OpenDAP do for me
o Benefits of Using OpenDAP
o Accessing data via OpenDAP (client)
o Serving your data via OpenDAP
    (provider)
o Some background on OpenDAP
Frequently Asked Questions             http://www.opendap.org/faq/index.html
OpenDAP Support                        http://www.opendap.org/support/index.html
o Quick Start Guide
o Server Installation Guide
o User Guide
o E-mail list
OpenDAP Software                       http://www.opendap.org/developers/third_party_software.html
o Clients
o Client-Libraries
o Servers
o Development kits




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                  15
                                                                         Guidelines for Data Provider
                                                     Quick OpenDAP Installation & Configuration Guide



2.3      Our Experience (AS versus GDS)

Data providers’ experience covers two type of OpenDAP servers, the Aggregation Server (AS) and the
GrADS Data Server (GDS):

                                                                                                      7
 AS has been used for the MERSEA Strand-1 intercomparison experiment                            (see
  http://strand1.mersea.eu.org/html/strand1/running.html ) and is operationnally used for instance to
  distribute satellite altimetry data (http://opendap.aviso.oceanobs.com – CatalogAggServer/0.6) or
  Foam         (http://www.nerc-essc.ac.uk:9090/dodsC/      --    CatalogAggServer/0.8),     Mercator
  (http://opendap2.mercator-ocean.fr            --      CatalogAggServer/0.6)        and         MFS
  (http://thredds.sincem.unibo.it:8080/thredds/dodsC/ -- CatalogAggServer/0.9) ocean forecasts.



 GDS is operationally used for instance to distribute in situ data and remote sensing sea surface
  temperature data at Ifremer (http://www.ifremer.fr/dodsG/ -- GrADS-DODS Server 1.2.9).




First feedback are:

   OpenDAP software are available for download without licensing costs.

     Servers are typically available as binary executables that the data provider can quickly install
      (first baseline for the OpenDAP implementation plan). Existing and new client applications are
      supported through the OpenDAP client libraries (second baseline for the OpenDAP
      implementation plan).

     The OpenDAP project provides the core of the system, a server software , written in C++ and
      Perl. Main developers are from the Graduate School of Oceanography at the University of Rhode
      Island, This software, called CGI base server software, aims at serving multi-dimensional data
      arrays and their meta-data. It comes with a set of six file format handlers, for serving different
      format of files, among which NetCDF and HDF4.

     Note: another server software from the same source, called SQL database server, and intended
      to serve data from a Relationnal Database, is not to be considered, even though it is conformant
      to OpenDAP protocol.

     We din’t try user authentification to access the data neither user profiles for data sets access, but
      only tested a restricted access to the Internet address

     Many OpenDAP servers support compression of the returned data. On the client side, there is a
      configuration option to enable compression in the local initialization file ($HOME/.dodsrc)




7
    OpenDAP Aggregation Server (AS, currently beta release, v0.6) has been used within Mersea Strand-1 it for
    temporal aggregation, creating thus a time serie data set accessible by a unique URL to the users - with or
    without subsetting characteristics (like geographical extraction, one variable) -.




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                              16
                                                                         Guidelines for Data Provider
                                                     Quick OpenDAP Installation & Configuration Guide

   OpenDAP Aggregation Server (AS) is the right tool to “aggregate” multiple datasets into one
    (OpenDAP) (virtual) dataset. The server works with local netCDF file as well as data accessible
    from a DAP-compliant server.

    Aggregation is particularly useful for large collections of files (temporal aggregation) that is when a
    long time series is split over several files, typically when, for each time step as the time pass, a new
    file is created.

     The Aggregation Server can be used as a “netCDF file” server, an “aggregation dataset” server,
      or a combination of both. The client will not see a difference between the two (except perhaps in
      performance). An aggregation dataset looks like a normal OpenDAP dataset to the client, but is
      actually a logical view of a group of other datasets. As currently implemented, the underlying
      datasets may be either netCDF files (local to the AS) or datasets obtained from other OpenDAP
      servers, called OpenDAP datasets.

     AS is a Java servlet (that is, software written in Java), builds on the Java/DODS Server library,
      and uses an XML configuration file to define the datasets that it can serve, as well as the extra
      information needed to specify how the aggregation is done (when the dataset is an aggregation
      dataset). This XML configuration file is based on the Thredds catalog. It specifies what datasets it
      serves, and how to aggregate them. The Thredds output makes the data and metadata more
      accessible to a greater variety of clients.


   GDS (GrADS Data Server), developed at COLA, makes data available via OpenDAP using
    GrADS (Grid Analysis and Display System), an analysis and visualization software package,
    also developed at COLA. The types of data that can be served include GRIB, BUFR, netCDF, HDF,
    and GrADS binary. It has also some aggregation capabilities.

    In addition to supporting the OpenDAP subsetting protocol, the GDS has a powerful server-side
    analysis capability. Analysis operations include basic math functions, averages, smoothing,
    differencing, correlation, and regression.


     As the Aggregation Server, GDS is a Java servlet that builds on the OpenDAP server mentioned
      above. It uses also an XML configuration file to define the datasets and the metadata that it can
      serve.

     Because all data requests in the GDS are passed to GrADS, and GrADS is itself an OpenDAP
      client, it was a natural step to allow the server administrator to provide URLs for remote
      OpenDAP datasets, as well as local file names, in the GDS configuration file. The GDS thus can
      act as an intermediary to other OpenDAP data servers (that is a gateway to remote OpenDAP
      datasets).

     The GDS supports operations that can be expressed in a single GrADS expression. Analysis
      tasks using GrADS functions can be encoded into URLs, which users can then pass into any
      OpenDAP client application.




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                                 17
                                                                      Guidelines for Data Provider
                                                  Quick OpenDAP Installation & Configuration Guide

     AS versus GDS

                                        AS                                           GDS
                       local netCDF file                           local GRIB, BUFR, netCDF, HDF
                       data accessible from a DAP-compliant          file
         Data
                        server                                      GrADS binary data are OpenDAP
        Served
                       easy data sets configuration with             readable
                        aggregation capabilities                    some aggregation capabilities
                       slightly different in the way they list     slightly different in the way they list
          Web
                        their datasets                                their datasets
       Interfaces
                       cf. example given above                     cf. example given above
                       need to link client applications            includes      server-side      analysis
       Analysis
      Capabilities     among them the LAS or Maltlab                 capability
                        toolbox
                       Apache log                                  includes      server-side      analysis
        Log
     Management        Note: Can be analysed with Analog             capability

                       still beta prototype, prefer AS version     GDS version 1.3 to be used with
                        0.6 (sept 2003) and not above                 GrADS version 1.8 or higher
                       easy operational configuration (to          includes a number of parameters to
                        specify what datasets are served, and         control the performance and
                        how to aggregate them)                        resource usage of the server (cf.
        Version
     Configuration     possibility to configure a permanent          maximum       simultaneous       client
                        cache to keep often requested                 connections, CPU time available
     Performances
                        products     and     increase   system        per analysis task, maximum
                        performance                                   requestable subset size, disk space
                       tested at Hycom US the fact of                for caching of analysis results)
                        developping queue job for heavy data
                        request
                       poor management of software versions        problem’s in attribute’s value
                        (but still in a beta mode)                   transmission
                       avoid large configuration file (above       some multi-dimensional request
                        90000 lines), use include instead            failures
                       poor error management, a lot of             do not handle grids with un-regular
                        ”unknown error”                              time axes
                                                                    heavy configuration for grid with
    Troobleshooting  check upon latest releases for software        unregular lat-lon steps (need of
                      enhancements and bug fixes                     data descriptor files)
                                                                    satellite swath data is not explicitly
                                                                     supported

                                                                   check upon latest releases for
                                                                    software enhancements and bug
                                                                    fixes
                       Administrator -- John Caron (Unidata)       Administrator -- Jennifer M. Adams
     Development       Request for funding, good hope within        (Cola)
          &             the scope of DAP2 and DMAC                  Funding provided by the SIESIP
     Maintenance                                                     grant from NASA's Earth Science
                                                                     Information Partnerships.



         Next part of this document will give our feedback on those two software




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                           18
                                                                       Guidelines for Data Provider
                                                 Quick OpenDAP Installation & Configuration Guide
       for installation, configuration or for routine operations like monitoring, log-file

       management, synchronization with a front-end server such as the Live Access

       Server ...




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                  19
                                                                        Guidelines for Data Provider
                                                    Quick OpenDAP Installation & Configuration Guide



3. GUIDELINES FOR OPENDAP INSTALLATION &
   CONFIGURATION


3.1     Pre-Requesites (Technical Requirements)
In this chapter, we present the required computer platform characteristics before installation of either the
AS or GDS.


3.1.1 Hardware
Information along this chapter assumes installation is done on an Intel-like Personal Computer, though it
is not a requirement by itself.

3.1.2 Operating System

OpenDAP software must be installed on a Unix-like operating system (including Linux, Sun-OS,
HP-UX,…)


3.1.3 Software
Both OPeNDAP server software require some external software to be installed first. All these software are
open-source and freely available.

 See information given for each Opendap servers to see needed software versions.

                                   AS                                            GDS

                AS requires the Java 2 Standard               GDS requires a Java Runtime
                 Edition (J2SE) SDK (Software                   Environment that is a Java Virtual
                 Development Kit) (not the JRE - Java           Machine (JVM)
                 Real-time Environment -).
                                                               Enter java -version at the Unix
    Java                                                        command prompt to find out what JVM
                                                                you have currently installed on your
                                                                system. The Java Virtual Machine is a
                                                                free download either from Sun
                                                                Microsystems, or your operating system
                                                                manufacturer's website.


                AS requires to install Tomcat before its      GrADS data server distribution includes
                 installation.                                  Tomcat.

                Installation of AS has been tested            Installation of GDS has been tested
  Tomcat         successfully with Tomcat 3.3, 4.x and          successfully with Tomcat 3.3 and 4.x.
                 5.x.

                We recommend reading the following
                 document:
                  o http://jakarta.apache.org/tomcat/to




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                             20
                                                                         Guidelines for Data Provider
                                                     Quick OpenDAP Installation & Configuration Guide
                       mcat-5.5-doc/setup.html
                   o   http://jakarta.apache.org/tomcat/to
                       mcat-5.5-doc/appdev/index.html

                 For downloading Tomcat 5.x, go to the
                  following page:
                  http://jakarta.apache.org/site/download
                  s/downloads_tomcat-5.cgi and
                   o For a Linux system, select the
                        binary .zip or tar.gz file
                   o For other Unix system, select the
                        source distribution zip or tar.gz.

                  It is recommended that you get some
                  help from your system administrator for
                  installing Tomcat.


               Apache http server is not required since Tomcat server is itself an http server but, for
                security reasons, use of Apache as a front-end http server is recommended. Use of
                Apache 2.0 is recommended.

               Apache server software can be downloaded from http://httpd.apache.org/. Information
                can be found in the FAQ page and in the comprehensive documentation.

               On most Linux platforms, Apache server is already installed. To check whether you
    Apache
                have it or not, type in the command – httpd – as user “root” or check for the
                presence of directory /etc/httpd/ on your computer.

               It is recommended that you get some help from your system administrator for
                configuring and starting Apache server.

               The information for configuring the communication Apache/Tomcat can be found
                here.


                 NetCDF                                        GrADS (Grid Analysis and Display
    Other        Enabling access via telnet, ftp, http, rsh     System) available
                  (an user account for each)                    NetCDF




3.1.4 Software Versions and Environment
    Prior to any installation check on the needed software versions and the one given the best
     performances.

    Do not forget to update your CVS environment when uploading new software.

    Do not forget to include this new technology in the back-up system (e.g. software version, data sets
     served, configuration files).

    To routinely operate your OpenDAP environment see some elements given in section 4.




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                             21
                                                                        Guidelines for Data Provider
                                                    Quick OpenDAP Installation & Configuration Guide




3.2     Aggregation Server (AS)

3.2.1 Introduction

   Prototype server since February 2002 (version 0.x)
   The Aggregation Server can be used as a “netCDF file” server, an “aggregation dataset” server, or a
    combination of both. We will see here the configuration for aggregation.
   The AS uses an XML configuration file (based on the Thredds catalog) to define the datasets that it
    can serve, as well as the extra information needed to specify how the aggregation is done (that is
    when the dataset is an aggregation dataset).

   The Thredds output makes the data and metadata more accessible to a greater variety of clients.
    That is the AS can show its Dataset Inventory Catalog on request, which is the "public view" of its
    available datasets.




3.2.2 Documentation & Contacts

   OpenDAP Aggregation Server Guide -- Aggregation, Installation, Configuration, Configuration
    Elements and Attributes Specification, THREDDS Catalog Configuration Elements

   OpenDAP User Guide -- Overview, Using OpenDAP to Read Data, the OpenDAP Client, Data
    Analysis with OpenDAP, The OpenDAP Server, Data and Data Models

   Troubleshooting or support needed -- send an email to John Caron or the dods-tech mailgroup

   The whole Opendap documentation with the OpenDAP Quick Start Guide




3.2.3 Download & Installation


                                       AS version 0.6 (sept 2003)
                                       to be used with Java 1.4.



   Access pages with Installation and Configuration information. Read the prior to any installation

   Installation is done simply by copying one file, thredds.war, on your system (in the directory
    $Tomcat_home/webapps). This file may be downloaded from here.

     When Tomcat starts up, it will see the thredds.war file and unpack it.

     With Tomcat 5.x, the web application thredds will be deployed automatically (directory thredds/
      created).

   The Tomcat engine is a stand-alone http server. It listens on one particular port, which is, by default
    after installation, port number 8080. This means that a user will access the main page of your server



Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                            22
                                                                        Guidelines for Data Provider
                                                    Quick OpenDAP Installation & Configuration Guide
    with a URL of the following type:
    http://www.my_server_name.my_domain_name:8080/thredds/dodsC/

     If you ever want to change this port number, you will find it in Tomcat main configuration file (an
      XML file): $TOMCAT_HOME/conf/server.xml

     The start/stop script ($tomcat_home/tomcat) can be modified to include some options, which can
      be very useful to increase memory for example. See for example appendix A.

    If you use Apache as a front-end http server, as it is recommended, this information is important since
    the Apache server will have to know where (on which port) to redirect http requests. The information
    for configuring the communication Apache/Tomcat can be found here.

   Do not forget to enable an access via telnet, ftp, http, rsh (an user account for each) and to organise
    the local storage to receive the data sets.



3.2.4 Configuration Information

3.2.4.1 Configuration Example For Serving one File

<!DOCTYPE catalog SYSTEM "http://www.unidata.ucar.edu/projects/THREDDS/xml/AggServerCatalog.dtd">
<catalog name="Mercator DODS Aggregation Server Catalog" version="0.6"
xmlns="http://www.unidata.ucar.edu/thredds">
    <dataset dataType="Grid" name="Mercator DODS Aggregation Server Catalog Top-level dataset"
serviceName="this">
        <service base="" name="this" serviceType="DODS"/>
        <service base="http://opendap-dev-
mercator.cls.fr/data/mercator_psy2v1_mersea_grid1o8_dc_med_tsuv/" name=" Exploit_Lev"
serviceType="NetCDF"/>

<dataset name="MERSEA_MED_LEVITUS_2001_00an1">
            <property name="internalService" value="Exploit_Lev"/>
                <dataset name="MERSEA_MED_LEVITUS_2001_00an1"
urlPath="MERSEA_MED_LEVITUS_2001_00an1.nc"/>
        </dataset>
<!--/////////////////////////////// End of file ////////////////////////////////-->
    </dataset>
</catalog>



3.2.4.1 Configuration Example For Temporal Aggregation

<!DOCTYPE catalog SYSTEM "http://www.unidata.ucar.edu/projects/THREDDS/xml/AggServerCatalog.dtd">
<catalog name="Mercator DODS Aggregation Server Catalog" version="0.6"
xmlns="http://www.unidata.ucar.edu/thredds">
    <dataset dataType="Grid" name="Mercator DODS Aggregation Server Catalog Top-level dataset"
serviceName="this">
        <service base="" name="this" serviceType="DODS"/>
        <service base="http://opendap-dev-
mercator.cls.fr/data/mercator_psy2v1_mersea_grid1o8_nat/" name=" Exploit_nat"
serviceType="NetCDF"/>
        <dataset name="Mercator_Best_Estimate_Atlantic"
urlPath="mercator_psy2v1_mersea_grid1o8_nat_best_estimate">
            <metadata metadataType="Aggregation">
                <aggregation serviceName="Exploit_Nat" aggType="JoinNew" varName="time"
varType="int" varUnit="days since 1950-01-01 00:00" dateFormat="yyy/M/d:HH:mm:ss z">
                    <fileAccess
urlPath="mercator_psy2v1_mersea_grid1o8_nat_b20040114_f200401019999.nc"
coord="2004/01/01:00:00:00 GMT"/>
                    <fileAccess
urlPath="mercator_psy2v1_mersea_grid1o8_nat_b20040114_f200401029999.nc"
coord="2004/01/02:00:00:00 GMT"/>




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                            23
                                                                  Guidelines for Data Provider
                                              Quick OpenDAP Installation & Configuration Guide
                    <fileAccess
urlPath="mercator_psy2v1_mersea_grid1o8_nat_b20040114_f200401039999.nc"
coord="2004/01/03:00:00:00 GMT"/>
                    <fileAccess
urlPath="mercator_psy2v1_mersea_grid1o8_nat_b20040114_f200401049999.nc"
coord="2004/01/04:00:00:00 GMT"/>
                    <fileAccess
urlPath="mercator_psy2v1_mersea_grid1o8_nat_b20040114_f200401059999.nc"
coord="2004/01/05:00:00:00 GMT"/>
                    <fileAccess
urlPath="mercator_psy2v1_mersea_grid1o8_nat_b20040114_f200401069999.nc"
coord="2004/01/06:00:00:00 GMT"/>
                    <fileAccess
urlPath="mercator_psy2v1_mersea_grid1o8_nat_b20040121_f200401079999.nc"
coord="2004/01/07:00:00:00 GMT"/>
                    <fileAccess
urlPath="mercator_psy2v1_mersea_grid1o8_nat_b20040121_f200401089999.nc"
coord="2004/01/08:00:00:00 GMT"/>
                </aggregation>
            </metadata>
        </dataset>
     </dataset>
</catalog>




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                             24
                                                                          Guidelines for Data Provider
                                                      Quick OpenDAP Installation & Configuration Guide



3.3     GrADS Data Server (GDS)

3.3.1 Introduction

   Officially born in September 2001 (version 1.0)

   GDS makes data available via OpenDAP using GrADS (Grid Analysis and Display System), an
    analysis and visualization software package. The types of data that can be served include GRIB,
    BUFR, netCDF, HDF, and GrADS binary. It has also some aggregation capabilities

   As the AS, GDS uses also an XML configuration file to define the datasets and the metadata that it
    can serve.

   In GrADS (and in GDS), the raw binary data and the meta data (information about the binary data)
    are stored in separate files. The meta data file contains a complete description of the binary data as
    well as instructions for GrADS on where to find the data and how to read it.

   Non-standard netCDF files require a data descriptor file that provides the metadata that GrADS
    requires to extract the data properly and place the grids in a lat/lon/lev/time context. Aggregation of
    data files also requires a data descriptor file.

   Both Tomcat and GDS support IP-address-based security. Each can be given their own security
    settings in their individual configuration files. To allow or deny access to specific GDS data sets and
    server features (e.g. download size limits), you can use the <ip-range> and <privilege> tags in the
    GDS configuration file.




3.3.2 Documentation & Contacts


   Copyright -- GNU public license

   Administrator's Guide -- Installation, Controlling the Server, Configuration, Adavnced Topics

   GDS/GrADS User's Guide --Accessing data from a web browser, Accessing data from a DODS-
    enabled analysis tool, Using remote data in scripts

   Configuration File Reference

   Troubleshooting or support needed -- post a message to the gradsusr mailing list or send an email to
    www@cola.iges.org

   The whole GDS documentation.




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                          25
                                                                           Guidelines for Data Provider
                                                       Quick OpenDAP Installation & Configuration Guide




3.3.3 Download & Installation



                       GDS version 1.3 (see readme file8) plus GrADS version 1.9
                                  to be used with Java 1.3 or higher




   The latest version of the GDS is available at the GDS home page as a compressed tar archive
    whereas GraDS is available here.
     Any number of GDSes can be run on the same system, as long as they are configured to use
       different ports (see Tomcat settings).

   After unpacking the archive,

     Edit the configuration file, and tell the GDS where to find GrADS, by editing the <invoker> tag.

     If you are not using a full GrADS distribution, make sure the GrADS executable you specify is
      capable of opening the types of dataset you wish to use.

   Next, double check with other users and/or your system administrator, to make sure that port settings
    for the GDS do not conflict with ports that are already in use. By default the GDS uses ports 9090 and
    9095. A user will access the main page of your GDS with a URL of the following type (because
    Tomcat          is          included            with         a          particular       configuration:
    http://www.my_server_name.my_domain_name:9090/dods/


     If you ever want to change this port number, you will find it in Tomcat main configuration file (an
      XML file): $GDS_HOME/tomcat/conf/server.xml.

     And read Tomcat settings for instructions on how to change these.

    If you use Apache as a front-end http server, as it is recommended, this information is important since
    the Apache server will have to know where (on which port) to redirect http requests. The information
    for configuring the communication Apache/Tomcat can be found here.

    At this point, you should be able to start the server and view the example dataset. You do not need
     root user access to run the GDS. There is no build or systems install process, because it is a cross-
     platform Java application.

     If you plan to serve netCDF, HDF, or DODS data, also make sure that the GADDIR environment
      variable in the GDS startup shell points to the location of the GrADS supplementary data files
      (available from the GrADS download page). In particular, the file udunits.dat must be present
      in this directory, since it is needed for COARDS metadata processing.

   Do not forget to enable an access via telnet, ftp, http, rsh (an user account for each) and to organise
    the local storage to receive the data sets.




8
    The Java DAP class library has been upgraded to version 1.1.7 which implements DAP 2.0. The Thredds XML
    catalog defaults to version 1.0. GDS version 1.3 also features a complete overhaul of attribute handling, with
    additional metadata for gridded and station data sets. The organization of environment variables and code
    relating to disk paths has been updated to support various server resources .



Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                                 26
                                                                      Guidelines for Data Provider
                                                  Quick OpenDAP Installation & Configuration Guide
3.3.4 Configuration Information


   Configuration

   GrADS Gridded Data Sets and the Data Descriptor File


3.2.4.1 GDS main Configuration File Example


See appendix B for an example of GDS main configuration file (gds.xml).



3.2.4.2 Data Descriptor File Example


Below is an example of a data descriptor file for an aggregation of netCDF data. The names of the data
file must match the pattern
/data/mercator/tomcat/webapps/dodsC/data/mercator_psy2v1_mersea/med/%y4%m2%d2.nc
like for example
/data/mercator/tomcat/webapps/dodsC/data/mercator_psy2v1_mersea/med/20050601.nc,
where the date of the file is parsed in the name.
If your files are not stored like this, you can create unix symbolic links.


DSET /data/mercator/tomcat/webapps/dodsC/data/mercator_psy2v1_mersea/med/%y4%m2%d2.nc
OPTIONS template
XDEF 341 LINEAR -6 0.125
YDEF 129 LINEAR 30 0.125
ZDEF 8 LEVELS 5 30 50 100 200 500 1000 2000
TDEF 588 LINEAR 22may2003 1dy
UNDEF 1.e+35 _FillValue
DTYPE netcdf
VARS 12
temperature 8 z,y,x temperature [degC]
salinity 8 z,y,x salinity
u 8 z,y,x zonal velocity [m/s]
v 8 z,y,x meridian velocity [m/s]
ssh 0 y,x ssh [m]
mld 0 y,x mld [m]
mlp 0 y,x mlp [m]
bsfd 0 y,x bsfd [m3/s]
taux 0 y,x taux [Pa]
tauy 0 y,x tauy [Pa]
qtot 0 y,x qtot [W/m2]
emp 0 y,x water flux [Kg/m2/s]
ENDVARS




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                           27
                                                                       Guidelines for Data Provider
                                                   Quick OpenDAP Installation & Configuration Guide



4      ROUTINE OPERATIONS
       WRT MERSEA INTEGRATED SYSTEM

Routine operations for individual OpenDAPs are needed with respect to the MERSEA Integrated System
that is system federation so we reached an operational environment.

We listed two of them a monitoring function and synchronization process for instance to link individual
Opendaps to a federating front-end server like the Live Access Server.

Simultaneously with the apparition of time series data sets, there is a need for a new concept, to define
and routinely update a Context File (History File) for each product line which will comment on time series
data set time evolutions, and report on operational product generation (cf. input data, configuration
parameters).




4.1     Monitoring Function

4.1.1 Introduction

To have operational systems, we identified two needs here:
 Regularly update your OpenDAP data server with new data sets released,
 Regularly check web site availability for automatic servers restart when needed.

To help each partner have a general view on the data sets availability and environment, we implemented
a status of each declared OpenDAP (cf. http://w3.mersea.eu.org/html/information/data_access.html) and
made it available in the MERSEA web site (http://bulletin.mersea.eu.org/).



4.1.2 Routine Opendap Update

   Regularly update your OpenDAP data server with new data sets released and expose them on
    Internet by simultaneously upgrading the configuration file.

   The operational server configuration is suited to the dissemination service’s strategy like a full
    temporal window versus a 6-month sliding window for instance, serving basic time series products or
    the best estimate concept for ocean forecasts.

   Within Mersea integrated system, there is a need to federate individual OpenDAPs. This need
    derived in a technical need that is simultaneously to the Opendap update, to generate and post a
    notification file so the Mersea central web site can synchronised itself. See section 4.2 for more
    information on this process.



4.1.3 Regularly Check Web Server Availability


   Web server must be frequently and automatically verified by a tool, which must be able to restart
    Tomcat and Apache servers if there is a problem.



Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                         28
                                                                        Guidelines for Data Provider
                                                 Quick OpenDAP Installation & Configuration Guide
   To serve this purpose, a perl program have been developed to do automatically (Unix crontab) these
    actions:
     Verify that the server send a good response to a request (access to metadata for example if it is
        an OpenDAP server)
     If the response is in error : send a mail to the responsible of the server, restart Tomcat and
      Apache server
     If the restart is not sufficient, some mails will be send to the responsible as long as the server
      doesn’t work.

   This perl program is available in appendix C: “ A Perl program to check regularly upon web server
    availability”. Some parameters are defined in a parameter file (see appendix D: “Parameter File for
    “verify_web.pl”), and two library files are used. All the files are available here (in a tar format). Please
    contact webmaster-mersea@mersea.eu.org for more explanation.


4.1.4 An Application: the OpenDAP Data Status

To help each partner have a general view on the data sets availability and environment, we implemented
a status of each declared OpenDAP (cf. http://w3.mersea.eu.org/html/information/data_access.html) and
made it available in the MERSEA web site (http://bulletin.mersea.eu.org/).

   This status is created for users who want to know which server is available, and the temporal window
    of the different data sets.

   The program creating the status interrogate, on a daily basis, each Opendap server for information
    about data sets served.




4.2     Synchronisation Process

4.2.1 Introduction

Within Mersea integrated system, there is a need to federate individual OpenDAPs.

   This need derived in a technical need that is simultaneously to the Opendap update, to generate and
    post a notification file so the Mersea central web site(s) can synchronised itself.


4.2.2 Synchronisation Process


   Each time a data provider will update its OpenDAP, it has to generate a notification file.

   The notification file is a formatted ‘xml’ file, which is put on an ftp site (identified by its released date)
    to communicate the entrance of new data. There is an ftp site for each data provider. Please send ftp
    information to webmaster-mersea@mersea.eu.org when it is available.

   The notification file should pass exhaustive information on the data sets served and the current
    OpenDAP configuration so for instance a central Mersea Live Access Server or any other tools for
    data manipulation and visualisation will be able to synchronised and update its list of data sets with
    appropriate new aggregation (time series served) or configuration parameters (especially for model
    output and the best estimate time series).




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                                 29
                                                                          Guidelines for Data Provider
                                                      Quick OpenDAP Installation & Configuration Guide
   The format of the notification file is explained below :




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                     30
                                                                  Guidelines for Data Provider
                                              Quick OpenDAP Installation & Configuration Guide

<?xml version="1.0" encoding="UTF-8"?>
<!-- URL of the Opendap server (source) and LAS server -->
<mersea_update opendap_source="http://opendap.mercator-ocean.fr"
                            las_destination="http://las.mersea.eu.org">
    <header>
<!-- Dates of previous and current notification (creation of the notification file on ftp site) -
->
        <dates previous="20050518" current="20050525"/>
<!-- Person to contact if problem -->
        <mail_info>lucas.nouel@mercator-ocean.fr</mail_info>
        <commentary>Mise a jour las RUN du 20050525 EFFECTUEE</commentary>
    </header>
<!-- Number of dataset which must be synchronised-->
    <datasets nb="2">
<!-- Path of the dataset (without opendap server URL, this dataset is available here:
http://opendap.mercator-ocean.fr/dodsC/mercator_psy2v1_mersea_grid1o8_med_best_estimate) and
information about -->
        <dataset opendap_url_path="mercator_psy2v1_mersea_grid1o8_med_best_estimate" area="MED"
                       model="PSY2V1" grid_size="grid1to8" aggregation_type="best_estimate"
new="no">
<!-- Dates to be deleted, if any -->
            <dates_deleted nb="0"/>
<!-- Date to be added -->
            <dates_new nb="7">
                <date>20050601</date>
                <date>20050602</date>
                <date>20050603</date>
                <date>20050604</date>
                <date>20050605</date>
                <date>20050606</date>
                <date>20050607</date>
            </dates_new>
        </dataset>
        <dataset opendap_url_path="mercator_psy2v1_mersea_grid1o8_nat_best_estimate" area="NAT"
                       model="PSY2V1" grid_size="grid1to8" aggregation_type="best_estimate"
new="no">
            <dates_deleted nb="0"/>
            <dates_new nb="7">
                <date>20050601</date>
                <date>20050602</date>
                <date>20050603</date>
                <date>20050604</date>
                <date>20050605</date>
                <date>20050606</date>
                <date>20050607</date>
            </dates_new>
        </dataset>
        <dataset opendap_url_path="mercator_psy2v1_mersea_grid1o8_med_forecast" area="MED"
                       model="PSY2V1" grid_size="grid1to8" aggregation_type="forecast" new="no">
            <dates_deleted nb="0"/>
            <dates_new nb="1">
                <date>20050530</date>
            </dates_new>
        </dataset>
        <dataset opendap_url_path="mercator_psy2v1_mersea_grid1o8_nat_forecast" area="NAT"
                       model="PSY2V1" grid_size="grid1to8" aggregation_type="forecast" new="no">
            <dates_deleted nb="0"/>
            <dates_new nb="1">
                <date>20050530</date>
            </dates_new>
        </dataset>
   </datasets>
</mersea_update>

For more information please contact webmaster-mersea@mersea.eu.org.




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                 31
                                                                      Guidelines for Data Provider
                                                  Quick OpenDAP Installation & Configuration Guide




4.2.3 An application: the Federating Live Access Server

   A centralised Live Access Server (http://las.mersea.eu.org) has been created to allow users to plot
    map of all Opendap servers of the MERSEA project. This will be a unique portal for all the data
    available in MERSEA.

   LAS synchronisation is performed every day of the week. As a result of this daily interrogation, the
    new daily fields were automatically charged on the visualization software (LAS) to be ready for their
    manipulation and visualization on Internet

   In order to facilitate the analysis of this huge amount of outputs, automatic procedures will be
    developed to automatically generate plots in a GIF format file. Example from Mersea Strand-1, some
    intercomparison plots between ocean models are available at http://bulletin.mersea.eu.org/.




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                        32
                                                                       Guidelines for Data Provider
                                                   Quick OpenDAP Installation & Configuration Guide




                            Figure 1: Synchronisation elements
                           for Mercator Opendap with Mersea Las



4.3     The Context File

4.3.1 Introduction

Simultaneously with the apparition of time series data sets, there is a need for a new concept, to define
and routinely update a Context File (History File) for each product line which will comment on time series
data set time evolutions, and report on operational product generation (cf. input data, configuration
parameters).



4.3.2 Context File Format

   Context file are text files released by each system that describe changes away from the systems
    description

   First context files have been generated in the frame of Mersea Strand-1 for model output description.
    They list all changes that have occurred during the inter-comparison time period, June2003- March
    2004, to the ocean forecasting systems. Follow the link to access the history text file for FOAM,
    MERCATOR, MFS and TOPAZ.




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                         33
                                                                 Guidelines for Data Provider
                                             Quick OpenDAP Installation & Configuration Guide




                       Figure 2: On the left, Topaz history file
              On the righ,t data operational status for class 1, 2,3 and 4




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                            34
                                                                       Guidelines for Data Provider
                                                   Quick OpenDAP Installation & Configuration Guide



APPENDIX A: SCRIPT TO STOP/START TOMCAT

Note: Do not forget to change the absolute path to yours !

#!/bin/sh
#
# Startup script for Jackarta Tomcat
#
# chkconfig: 345 85 15
# description: Apache is a World Wide Web server.      It is used to serve \
#              HTML files and CGI.
# processname: tomcat

# Source function library.
. /etc/rc.d/init.d/functions

JAVA_HOME="/usr/java/j2sdk1.4.1"
PATH="$JAVA_HOME/bin:$PATH:/usr/X11R6/bin"
TOMCAT_HOME="/var/tomcat4/"
CATALINA_HOME=$TOMCAT_HOME
CATALINA_BASE="/data/mercat1/tomcat"
CLASSPATH=$CLASSPATH
TOMCAT_CFG="$CATALINA_BASE/conf/tomcat4.conf"
TOMCAT_OPTS="-Xmx512m"
JAVA_OPTS="-Xmx512m"
CATALINA_TMPDIR="$CATALINA_BASE/temp"
JAVADIR="/usr/java/j2sdk1.4.1/"

export JAVA_HOME TOMCAT_HOME PATH CLASSPATH TOMCAT_OPTS CATALINA_HOME CATALINA_BASE
CATALINA_TMPDIR JAVADIR TOMCAT_CFG JAVA_OPTS

# See how we were called.
case "$1" in
  start)
         echo -n "Starting tomcat: "
         cd $TOMCAT_HOME
         bin/startup.sh –config /data/mercat1/tomcat/conf/server.xml
         echo
         ;;
  stop)
         echo -n "Shutting down tomcat: "
         cd $TOMCAT_HOME
         bin/shutdown.sh –config /data/mercat1/tomcat/conf/server.xml
        # wait for shutdown
        sleep 2
        # check if some tomcat process still alive
        pids=`ps auxww | grep java | grep -v grep | grep "Dcatalina.base=$CATALINA_BASE " | awk
'{print $2}'`
        if [ -n "$pids" ]; then
                # if process alive, start a new delay
                sleep 5
                pids=`ps auxww | grep java | grep -v grep | grep "Dcatalina.base=$CATALINA_BASE "
| awk '{print $2}'`
                # if still alive, kill it
                if [ -n "$pids" ]; then
                        echo "Tomcat running force killing it"
                               kill -9 $pids
                fi
         fi
         echo
         ;;
  restart)
         $0 stop
         $0 start
         ;;
  status)
        pids=`ps auxww | grep java | grep -v grep | grep "Dcatalina.base=$CATALINA_BASE " | awk
'{print $2}'`
        if [ -n "$pids" ]; then
                echo "Tomcat running"
         else



Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                  35
                                                                   Guidelines for Data Provider
                                               Quick OpenDAP Installation & Configuration Guide
               echo "Tomcat not running"
          fi
         ;;
 *)
         echo "Usage: $0 {start|stop|restart|status}"
         exit 1
esac

exit 0




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                              36
                                                                  Guidelines for Data Provider
                                              Quick OpenDAP Installation & Configuration Guide



APPENDIX B: GDS MAIN CONFIGURATION FILE

Note: Do not forget to adapt file name !

<?xml version="1.0" encoding="ISO-8859-1"?>
<!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
This is a configuration file for the GrADS-DODS Server.
See README and doc/index.html for general GDS information.
See doc/tag-ref.html for help editing this file.
If you want to revert your GDS to its "factory settings", overwrite
gds.xml with the file gds.xml.reference.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
<gds>
  <catalog temp_size_limit="0">
    <data>
      <dataset name = "mercator_psy2v1_mersea_grid1o8_med"
        file="mercator_psy2v1_mersea_grid1o8_med.ctl"
        format="nc">
    </data>
  </catalog>

  <log mode="file" file="log/gds.log" level="debug" />
  <grads>
     <invoker grads_dir="/data/mercator/grads-1.9b3/bin"/>
     <!-- edit this tag to point to your GrADS distribution -->
  </grads>
  <servlet>
    <filter-abuse enabled="false" hits="1000" timeout="24" />
    <filter-overload enabled="false" limit="20" />
    <!-- edit these tags to enable 'throttling' features -->
    <filter-analysis enabled="true" />
  </servlet>
  <mapper>
    <service-admin enabled="false" auth="open-sesame" />
    <!-- edit this tag to enable the administration service -->
  </mapper>
  <privilege_mgr default="public">
    <ip_range mask="127.0.0.1" privilege="private" />
    <privilege name="public">
    <!-- edit this tag to control public access -->
      <deny path="private" />
    </privilege>
    <privilege name="private" /> <!-- no deny tags, allows full access -->
    <!-- specify more privilege sets and IP range assignments here -->
  </privilege_mgr>
</gds>




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                             37
                                                                       Guidelines for Data Provider
                                                   Quick OpenDAP Installation & Configuration Guide



APPENDIX C: A PERL PROGRAM
TO CHECK REGULARLY WEB SERVER AVAILABILITY

Note: Do not forget to change the absolute path to yours !

#!/usr/bin/perl
#
use LWP::UserAgent;
require "/data/mercator/surveillance_web/mail-lib.pl";
require "/data/mercator/surveillance_web/util.pl";

# Fichier de parametres
require "/data/mercator/surveillance_web/param_verify_web.pl";

$ua    = LWP::UserAgent->new();

@url = keys %sites;
foreach $url (@url) {

         $req = HTTP::Request->new( GET => $url );

         # Inclusion de user passwd pour l eventuelle authentification http
         $http_user = ${$sites{$url}}{'http_user'};
         $http_passwd = ${$sites{$url}}{'http_passwd'};

         $req->headers->authorization_basic( $http_user, $http_passwd ) if (!$http_user eq "");

       # Configuration du proxy si necessaire
       if (!$proxy eq "") {
               $ua->proxy(http => $proxy);
               $ua->env_proxy;
               # Authentification proxy
               $req->headers->proxy_authorization_basic($user_proxy, $pass_proxy ) if
(!$user_proxy eq "");
       }
       # Requete
        $res = $ua->request( $req );

         # Test du resultat
          if ($res->is_success) {
                  print "--------------------------------------- OK -------------------------------
\n";
                print "Le site $url a renvoye une reponse correcte.\n";
                # Flux html constituant la reponse sur la sortie standard
                 #print $res->content;
                 #print "\n-------------------------------------Fin-------------------------------
-\n"
         } else {

                print "\n*********************************** Error
******************************\n", $res->status_line, "\n";

                # redemarrage tomcat
                system "$commandStop";

                system "$commandStart";

                $date = &affDate;
                $titleMail = "WARNING : Erreur pour le site web $url";
                $content = "Current date : $date.\n";
                $content.= ${$sites{$url}}{'content'};

                @mail = @{${$sites{$url}}{'mail'}};
                foreach $mail (@mail) {
                       send_mail($adMailSender, $mail, $titleMail, $content);
                }

         }

} #foreach



Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                    38
                                                                  Guidelines for Data Provider
                                              Quick OpenDAP Installation & Configuration Guide



APPENDIX D: PARAMETER FILE FOR “VERIFY_WEB.PL”

########### Parameters program verify_web.pl #################
# Created by T.Jolibois 11/08/2004   (tjolibois@cls.fr)                                    #
#                                                                          #
# Be carefull with special caracters (@ for example), put a \ before
# Rq : attention au \ avant les lettres specials (@ dans une adresse mail par ex)#
################################ List of web site ################################
# hash table with URL, mail adresses of contact when there is an error, eventually
# http login/passwd, and content of text mail.

# Le hashage permet de renseigner l'url du site, les mails des personnes a contacter
# en cas d'erreur (attention au \ avant le @), eventuellement un login/passwd pour une
authentification http,
# et eventuellement un complement de texte (content) qui sera envoye dans le mail en cas
d'erreur.
# Pour chaque site, l'url est indispensable ainsi qu'un mail, pour le reste laisser ""
# si le parametre n'a pas d'utilite.
#################################################################################

%sites = (
        "http://opendap-dev-mercator.cls.fr/dodsC/"=> {"mail" => ["tjolibois\@cls.fr"],
                                                       "http_user" => "", # for the web site
which have http authentification
                                                       "http_passwd" => "",
                                                       "content" => "Put text of the mail here\n"
                                                         },
        "http://opendap.mercator-ocean.fr/dodsC/"=> {"mail" =>
["tjolibois\@cls.fr","sbaudel\@cls.fr"],
                                                     "http_user" => "", # for the web site which
have http authentification
                                                     "http_passwd" => "",
                                                     "content" => ""
                                                             }
);

################################## Tomcat command ###########################
# Where the tool can restart Tomcat server
$commandStop = "/data/mercator/tomcat/tomcat stop";
$commandStart = "/data/mercator/tomcat/tomcat start";

################################## mail server ##############################
# Mail server which allow to send mail
$smtp_server = 'mail.cls.fr';

################################# Configuration proxy ############################
# Proxy ; if the program is behind a proxy
$proxy = "";
# Authentification proxy
$user_proxy = "";
$pass_proxy = "";
# Mail sender, can be a non existent mail adress, just appears in "sender" when send a mail
$adMailSender = "surveil_web_mercator\@cls.fr";
# retourne vrai
return 1;




Ref.: aba034e4-fb2d-4c1e-91d0-aa5ebc629579.doc                                                 39

								
To top