Docstoc

AWIPS II SCOPE

Document Sample
AWIPS II SCOPE Powered By Docstoc
					                       AWIPS II System Administrator Training
                                                   Release 2



                Module 1. AWIPS II System
                             Architecture

                                                            Prepared Under
                                            Contract DG133W-05-CQ-1067
                   Advanced Weather Interactive Processing System (AWIPS)
                                                Operations and Maintenance

T0010 AWIPS II Training and Documentation Task Order, Work Assignment 003




                                                                       For:
                                              U.S. Department of Commerce
                           National Oceanic and Atmospheric Administration
                                                   National Weather Service
                                                      NWS Training Center


                                                                       By:


                                 Raytheon Technical Services Company LLC
                                            8401 Colesville Road, Suite 800
                                                  Silver Spring, MD 20910


                                                                 Delivered:
                                                               1 June 2011




                                           DCN AWP.TRG.TOTD.SA-02.00
AWIPS II SA Training, Release 2                                   NWSTC Training Guide




              Module 1. AWIPS II System Architecture

                                     OBJECTIVES
   Upon completion of this module, the Electronic Systems Analyst (ESA)/ Information
    Technology Officer (ITO) will understand the overall AWIPS II architecture.
   Upon completion of this module, the ESA/ITO will know where the AWIPS II
    software is deployed.
   Upon completion of this module, the ESA/ITO will have the knowledge to describe
    the patterns used in storage of raw and processed data in the AWIPS II system.



                                     RESOURCES
   AWIPS System Manager’s Manual Chapter 2 – AWIPS System Architecture
   AWIPS System Manager’s Manual Chapter 25 – AWIPS II/EDEX Administration
    Guide
   AWIPS System Manager’s Manual Appendix Q – AWIPS Applications and Version
    Numbers
   AWIPS System Manager’s Manual Appendix R – System Architecture Diagrams
   AWIPS System Manager’s Manual Appendix U – AWIPS II Component
    Categorization

                                  MODULE EXERCISES
   None




Lecture 1                                -i-                 AWIPS II System Architecture
AWIPS II SA Training, Release 2                                                               NWSTC Training Guide




                       AWIPS II SYSTEM ARCHITECTURE:
                                  CONTENTS
                                                                                                                    Page
ARCHITECTURE OVERVIEW ........................................................................................ 1
BASIC AWIPS II SOFTWARE DEPLOYMENT ............................................................. 2
 Reengineered/Rehosted/Wrapped Functionality ............................................................ 2
   What’s New in AWIPS II ............................................................................................... 3
   What’s Changed for AWIPS II ....................................................................................... 3
   What’s Still Used From AWIPS I................................................................................... 3
BASIC AWIPS II SOFTWARE DEPLOYMENT ............................................................. 4
AWIPS II SERVER CLUSTERING .................................................................................. 5
BASIC AWIPS II COMMUNICATION ARCHITECTURE ............................................ 7
BASIC AWIPS II LOGGING ARCHITECTURE ............................................................. 8
BASIC AWIPS II DATA STORAGE ARCHITECTURE................................................. 8
BASIC AWIPS II VISUALIZATION ARCHITECTURE ................................................ 9
BASIC AWIPS II DATA PROCESSING ARCHITECTURE ......................................... 10
 Basic AWIPS II Data Receipt Architecture .................................................................. 10
   Basic AWIPS II Data Decoding Architecture .............................................................. 11
   Basic AWIPS II Processed Data Storage Architecture ................................................. 13
   Python Process Isolated Exchange Storage (PyPIES) .................................................. 14
   Basic AWIPS II Data Retrieval Architecture ............................................................... 14
BASIC AWIPS II DATA RETRANSMISSION DESIGN .............................................. 15
BASIC AWIPS II DATA PURGE ARCHITECTURE .................................................... 16
 AWIPS II Processed Data Storage Purge Architecture ................................................ 16
   AWIPS II Raw Data Storage Purge Architecture ......................................................... 17
REVIEW QUESTIONS .................................................................................................... 18
OBJECTIVES REVIEW .................................................................................................. 21


                                         LIST OF EXHIBITS
Exhibit 1. AWIPS II WFO Deployment ............................................................................. 2
Exhibit 2. Deployment Diagram for AWIPS II .................................................................. 4
Exhibit 3. AWIPS II Inter-Process Communication ........................................................... 7
Exhibit 4. AWIPS II Visualization ..................................................................................... 9
Exhibit 5. AWIPS II Data Receipt .................................................................................... 10


Lecture 1                                                 - ii -                       AWIPS II System Architecture
AWIPS II SA Training, Release 2                                                                NWSTC Training Guide


Exhibit 6. AWIPS II Data Decoding................................................................................. 12
Exhibit 7. AWIPS II Processed Data Storage ................................................................... 13
Exhibit 8. AWIPS II Data Retrieval ................................................................................. 14
Exhibit 9. AWIPS II Message Retransmission ................................................................. 15
Exhibit 10. AWIPS II Processed Data Storage Purge ....................................................... 16
Exhibit 11. AWIPS II Raw Data Storage Purge ............................................................... 17


                                           LIST OF TABLES
Table 1. AWIPS II Software ............................................................................................... 5
Table 2. AWIPS II Server Failover Strategies .................................................................... 5
Table 3. AWIPS II Server Aliases ...................................................................................... 6
Table 4. Log Locations for AWIPS II Components ........................................................... 8
Table 5. AWIPS II Data Storage......................................................................................... 9
Table 6. AWIPS II Interfaces to Data Sources ................................................................. 11




Lecture 1                                                  - iii -                      AWIPS II System Architecture
AWIPS II SA Training, Release 2                                       NWSTC Training Guide




ARCHITECTURE OVERVIEW
AWIPS II replaces a significant amount of the functionality of the system while
introducing new paradigms for data decoding and storage, background data processing,
and data visualization.

AWIPS II installs new software on a number of existing AWIPS systems: CPSBN1 and
CPSBN2; PX1 and PX2; DX1–4; and the LX and XT workstations.

In addition to the new software, part of the existing software has been retained; this
software is referred to as “rehosted applications.” The rehosted server applications are
primarily located on PX1 and PX2.

A number of additional resources are available if you would like more information on the
AWIPS II System Architecture, including these chapters and appendixes of the AWIPS
System Manager’s Manual (SMM):

           Chapter 2, AWIPS System Architecture, presents an overview of the entire
            AWIPS system.
           Chapter 25, AWIPS II/EDEX Administration Guide, provides information on
            the installed location of AWIPS II software.
           Appendix R, System Architecture Diagrams, provides a view of the specific
            systems modified in the migration to AWIPS II.
           Appendix Q, AWIPS Applications and Version Numbers, provides a list of
            applications making up AWIPS II.
           Appendix U, AWIPS II Component Categorization, provides information on
            the migration of AWIPS functionality into AWIPS II.




Lecture 1                                   -1-                  AWIPS II System Architecture
AWIPS II SA Training, Release 2                                     NWSTC Training Guide




BASIC AWIPS II SOFTWARE DEPLOYMENT
Refer to Exhibit R.0-1, CONUS WFO Architecture (AWIPS II Components Deployment
Concept) in SMM Appendix R for a detailed diagram of the basic AWIPS II deployment.
For convenience, this diagram is reproduced here as Exhibit 1.




                            Exhibit 1. AWIPS II WFO Deployment


In Exhibit 1, AWIPS II-specific software is represented as small green boxes on the
affected servers. Specific details of the server hardware and network topology are
included in SMM Chapter 2.

Reengineered/Rehosted/Wrapped Functionality
It is important to note that AWIPS II software does not replace all of the existing AWIPS
functionality. AWIPS II primarily replaces the existing data ingest/storage and
visualization/forecasting functionalities. The AWIPS II implementation of functionality
has been classified as reengineered, rehosted, and wrapped functionality. These terms are
defined in Appendix U, AWIPS II Component Categorization, of the SMM.




Lecture 1                                  -2-                 AWIPS II System Architecture
AWIPS II SA Training, Release 2                                       NWSTC Training Guide



What’s New in AWIPS II
AWIPS II replaces much of the existing data processing and forecaster tools with the
Environmental Data Exchange (EDEX) and the Common AWIPS Visualization
Environment (CAVE). A discussion of the deployment and architecture for the new
AWIPS II software follows.

What’s Changed for AWIPS II
Changes made to the deployed software are covered in SMM Appendix U.
AWIPS II continues to use heartbeat software to manage application packages on the
PX1/2 and DX1/2 clusters. Although heartbeat is no longer used on the DX3/4 server
pair, it is being added to the CPSBN1/2 server pair. AWIPS II clustering strategy and
heartbeat packages are discussed in “AWIPS II Server Clustering,” in this module.
Because AWIPS II does not utilize the heartbeat cluster on DX3/4, any locally defined
crons that were formerly managed by the dx3apps and dx4apps packages will need to be
moved to another server.
Additional file system mounts to the Network Attached Storage (NAS) are required for
AWIPS II; in addition, there are new mounts on the Direct Attached Storage (DAS). File
system mounts are covered in “AWIPS II Data Storage Architecture,” in this module.
AWIPS II software performs its own logging; log locations vary with the different
AWIPS II software components. AWIPS II logging is covered in “AWIPS II Logging
Architecture,” in this module.

What’s Still Used From AWIPS I
AWIPS II reuses a number of the existing AWIPS I applications. These include:
           The Message Handling System (MHS)
           Climate software
           HWR
           NWRWAVES/NWREditor/CRS
           FSI (server path has changed; the GUI has been wrapped)
           Local Data Acquisition and Dissemination (LDAD)
           NWWS
           ASYNC Scheduler
           LSR
           FloodEvent Archiver
           HydroGen
           RiverPro
           WFO/RFC Archiver
           LAPS/MSAS
           Chatserver




Lecture 1                                  -3-                 AWIPS II System Architecture
AWIPS II SA Training, Release 2                                           NWSTC Training Guide


            LegalArchiver
            Certain Command Line Interfaces (textdb, fxaAnnounce,
             sendMsgToGuardian)
For the AWIPS I software that has been retained, basic management techniques are
unchanged; high-availability packages and server cron configuration files are still used
(the package filenames start with a2, for example a2px2apps). Log names and locations
remain the same.

BASIC AWIPS II SOFTWARE DEPLOYMENT
AWIPS II installs new AWIPS II-specific software on several of the existing AWIPS
systems, both servers and workstations. This deployment concept is shown in Exhibit 1.
The deployment of AWIPS II software is also shown in Exhibit 2, which includes only
the systems being modified for AWIPS II.



                                    DB                     IPVS              re-host

                                   RCM                     QPID
       DB Tables
        HDF5
                                  PyPIES                 LDM(U)
       Raw Data

                 DAS              LDM(D)
                                   bridge
                                         DX1/2                   CP1/2               PX1/2



                                                         AWIPS LAN


                                                 SEDA Load Balanced
            CAVE
            CAVE                    EDEX                               EDEX
                                  EDEX/GRIB
                                  EDEX/Request                       EDEX/GRIB
                                                                     EDEX/Request

                LX/XT                              DX3                               DX4



                        Exhibit 2. Deployment Diagram for AWIPS II


Table 1 lists the specific AWIPS II-provided software installed on each system.




Lecture 1                                    -4-                     AWIPS II System Architecture
AWIPS II SA Training, Release 2                                       NWSTC Training Guide



Table 1. AWIPS II Software
     System                              Key Software Packages
CPSBN1 and             LDM(U), QPID, Java, Python, rehost
CPSBN2
PX1 and PX2            Rehost Applications
DX1 and DX2            PostgreSQL Data Base Management System (DBMS), Python,
                       Java, PSQL, PyPIES, RCM, LDM(D), rehost
DX3 and DX4            EDEX, Java, PSQL, Python, Command Line Interface (CLI) tools
LX Workstations        CAVE, AlertViz, CLI, PSQL, Java, Python
XT Workstations        CAVE, AlertViz, CLI, PSQL, Java, Python

On the servers – CPSBN1 and CPSBN2, PX1 and PX2, and DX1–4 – AWIPS II software
is installed in /awips2. On the workstations, AWIPS II software is installed in
/usr/local/viz.

AWIPS II SERVER CLUSTERING
As a general rule, the AWIPS II server software is deployed on pairs of servers. This
approach is used to maximize processing availability; various failover strategies are
employed, as detailed in Table 2.

Table 2. AWIPS II Server Failover Strategies
 Server Pair                                    Failover Strategy
CPSBN1 and         CPSBN1 And CPSBN2 function as a heartbeat cluster; failover is
CPSBN2             managed automatically or manually. Note that only a single high-
                   availability package is currently used on this cluster. The package name
                   is a2cp1apps. In failover mode, package execution shifts to the backup
                   server. Note that there is software, notably the LDM(U) processes, that
                   runs on both CPSBN1 and CPSBN2 and is not part of the a2cp1apps
                   package. The a2cp1apps package manages the cp1f floating server
                   name. Note that floating names are available only when the high-
                   availability packages are running; they cannot be used to start or stop
                   the high-availability packages.
PX1 and PX2        PX1 and PX2 function as a heartbeat cluster; failover is managed
                   automatically or manually. Note that both PX1 and PX2 normally run
                   separate high-availability packages; a2px1apps on PX1 and a2px2apps
                   on PX2. In failover mode, both packages shift to the same server.
                   Access to PX1 and PX2 is via floating server names: px1f points to the
                   server running the a2px1apps package; px2f points to the server
                   running the a2px2apps package. Note that the floating server names are
                   available only when the high-availability packages are running; they
                   cannot be used to start or stop the high-availability packages.
DX1 and DX2        DX1 and DX2 function as a heartbeat cluster; failover is managed
                   automatically or manually. Note that both DX1 and DX2 normally run


Lecture 1                                   -5-                  AWIPS II System Architecture
AWIPS II SA Training, Release 2                                        NWSTC Training Guide



Table 2. AWIPS II Server Failover Strategies
 Server Pair                                   Failover Strategy
                   separate high-availability packages; a2dx1apps on DX1 and a2dx2apps
                   on DX2. In failover mode, both packages shift to the same server.
                   Access to DX1 and DX2 is via server name alias: dx1f points to the
                   server running the a2dx1apps package; dx2f points to the server
                   running the a2dx2apps package. Note that the floating names are
                   available only when the high-availability packages are running; they
                   cannot be used to start or stop the high-availability packages.
DX3 and DX4        DX3 and DX4 are independent servers, both of which run the EDEX
                   server software. There is no failover between DX3 and DX4. Both
                   servers are load balanced and share data processing responsibilities; if
                   one server is stopped, the other will pick up the entire data processing
                   load. Processing load balancing between DX3 and DX4 is managed by
                   QPID. Client request balancing for DX3 and DX4 is managed by IPVS
                   (the pulse service). Both QPID and IPVS run on CPSBN1 and
                   CPSBN2.

As indicated in Table 3, server name aliasing is used to direct clients to the appropriate
member of each cluster. “Appropriate member” generally refers to the server running a
specific high-availability package; in the case of DX3 and DX4, however, it refers to an
available EDEX process. AWIPS II server aliases and their target servers are listed in
Table 3.

Table 3. AWIPS II Server Aliases
 Server Alias                                    Target System
dx1f                The server, DX1 or DX2, which is currently running the a2dx1apps
                    high-availability package.
dx2f                The server, DX1 or DX2, which is currently running the a2dx2apps
                    high-availability package.
px1f                The server, PX1 or PX2, which is currently running the a2px1apps
                    high-availability package.
px2f                The server, PX1 or PX2, which is currently running the a2px2apps
                    high-availability package.
cp1f                The server, CPSBN1 or CPSBN2, which is currently running the
                    a2cp1apps high-availability package.
ec, edexcluster     Connections are redirected to DX3 or DX4, based on a least
                    connections strategy. In a least connections strategy, a new connection
                    request is forwarded to the server, DX3 or DX4, currently having the
                    lower number of connections. This alias only applies to connections
                    on port 9581. These connections are managed by IPVS service, which
                    is managed by the a2cp1apps high-availability package, which runs on
                    CPSBN1 or CPSBN2.



Lecture 1                                   -6-                  AWIPS II System Architecture
AWIPS II SA Training, Release 2                                      NWSTC Training Guide


Note that the floating names, e.g., dx1f, px1f, are managed by the high-availability
packages; as a result, the floating server names are only available when the packages are
running and cannot be used when starting or stopping the high-availability packages.

BASIC AWIPS II COMMUNICATION ARCHITECTURE
The AWIPS II ingest and request processes are a highly distributed system; messaging is
used for Inter-Process Communication (IPC). Exhibit 3 shows the basic IPC architecture.




                     Exhibit 3. AWIPS II Inter-Process Communication

Exhibit 3 shows the basic lines of communication that are used for IPC in AWIPS II.
There are three primary communication channels:
        1. Advanced Message Queuing Protocol (AMQP) messages are routed between
           the various AWIPS II processes via the QPID message broker.
        2. Transport Control Protocol (TCP) messages are routed between EDEX and
           other AWIPS II components.
        3. TCP messages from CAVE to EDEX are routed through the IPVS to request
           load balancing between DX3 and DX4.
Combined with the clustering strategies discussed in “AWIPS II Server Clustering,” the
Communication Architecture provides improved system availability and processing load
balancing.




Lecture 1                                  -7-                  AWIPS II System Architecture
AWIPS II SA Training, Release 2                                     NWSTC Training Guide



BASIC AWIPS II LOGGING ARCHITECTURE
AWIPS II does not implement a standardized logging strategy; rather, each AWIPS II
component provides process logs that may be examined to determine process status and
assist in system troubleshooting. The log locations for the AWIPS II components are
listed in Table 4.

Table 4. Log Locations for AWIPS II Components
  Server                Process                            Log Directory
dx1f           Radar Server                 /awips2/rcm/data/logs
               PostgreSQL DBMS              /awips2/data
                                            /awips2/data/pg_log
               PyPIES                       /awips2/pypies/logs
                                            /awips2/http_pypies/var/log/httpd
                                            /tmp
dx2f           LDM(D)                       /usr/local/ldm/logs
               scour                        /usr/local/ldm/logs
               cron                         /var/logs
                                            Note: logs to cron log.
dx3/dx4        EDEX                         /awips2/edex/logs
cpsbn1/        QPID                         /awips2/qpid/var/log
cpsbn2                                      /var/logs
                                            Note: logs to message log.
               LDM(U)                       /usr/local/ldm/logs
LX/XT          AlertViz                     ~/caveData/logs
               CAVE                         ~/caveData/etc/user/${USER}/logs

For more information on AWIPS II process logging, including log naming patterns and
configuration, see the applicable sections of Chapter 25 of the SMM.

BASIC AWIPS II DATA STORAGE ARCHITECTURE
AWIPS II stores on two shared data servers – the Direct Attached Storage (DAS) server
and the Network Attached Storage (NAS) server. Both of these servers provide for shared
data access; this provides improved system resiliency in the event of server failure. DAS
access is limited to the DX1/2 cluster; NAS access is available from any server or
workstation on the Local Area Network (LAN), although not all systems are mounted to
access the data on the NAS. The AWIPS II Raw Data Storage is directly accessible only
from dx2f; however, other systems, notably DX3 and DX4, also access this data. They do
so using a file system map to dx2f:/data_store. The primary AWIPS II data storage
locations are listed in Table 5.




Lecture 1                                 -8-                  AWIPS II System Architecture
AWIPS II SA Training, Release 2                                         NWSTC Training Guide


Table 5. AWIPS II Data Storage
Data Component                            Location                Access From
AWIPS II Raw Data Storage                 das1:/aiildm            dx2f
AWIPS II HDF5 Data Storage                das1:/aiihdf5           dx1f
AWIPS II Database                         das1:/aiidb             dx1f
AWIPS II QPID Shared Data                 nas1:/qpid              cp1f
AWIPS II Static Data                      nas1:/aiidata           DX3 and DX4
AWIPS II Localization Storage             nas1:/aiidata           DX3 and DX4
AWIPS II Service Backup                   nas1:/GFESuite2         Various systems

Note that for mounts that are accessed from a floating server name, e.g., dx2f, the mounts
are managed by the high-availability package that provides the floating name. For
example, the mount to das1:/aiildm, which is accessed using the dx2f floater, is managed
by the a2dx1apps high-availability package.

BASIC AWIPS II VISUALIZATION ARCHITECTURE
AWIPS II uses the CAVE application as its main data visualization/manipulation tool.
CAVE incorporates a number of the legacy AWIPS I applications such as Display Two-
Dimensional (D2D), Graphical Forecast Editor (GFE), and Text Workstation. AWIPS II
has also reengineered GUARDIAN into AlertViz. CAVE and AlertViz are installed on
both the LX and the XT workstations. Exhibit 4 shows the basic visualization
architecture.


                                           IPVS                           2
                    1
                                          CPSBN1/2
                                                                       EDEX
              CAVE                              3                     Request
                                               4                        EDEX
            AlertVIZ                                                    Ingest
                     LX/XT                                                  DX3/4

                              Exhibit 4. AWIPS II Visualization

Exhibit 4 shows the basic information/data flow between CAVE and EDEX. AlertViz
will display CAVE errors as well as messages from other parts of the system. The
information/data flow is as follows:
        1. Data requests are sent from CAVE to a virtual server name (ec). The virtual
           server name is resolved by IPVS.



Lecture 1                                   -9-                    AWIPS II System Architecture
AWIPS II SA Training, Release 2                                           NWSTC Training Guide


        2. IPVS forwards the data request to the available EDEX Request process on
           either DX3 or DX4.
        3. The EDEX Request process returns the requested data to CAVE.
        4. The EDEX Ingest process broadcasts data-ingested messages that are picked
           up by CAVE.
CAVE can be started in various modes by passing its switches through the command line
(or via application launchers). For example, CAVE can be launched to emulate the Text
Workstation or only to display the D2D perspective. See SMM Chapter 25 for more
information on the CAVE command line switches.

BASIC AWIPS II DATA PROCESSING ARCHITECTURE
AWIPS II separates data processing into four logically separate components:
1) data receipt; 2) data decoding; 3) data storage; and 4) data request. These components
are distributed over the various systems in AWIPS II. This distribution is designed to
provide improved data throughput coupled with improved system resiliency. AWIPS II
reuses a number of existing components in data processing.

Basic AWIPS II Data Receipt Architecture
AWIPS II receives data from three main sources: the Satellite Broadcast Network (SBN);
the LDAD network; and the Open Radar Product Generator (ORPG)/Supplemental
Product Generator (SPG) network. Regardless of the data source, the received data is fed
to data ingest, which is performed by the EDEX software on DX3 and DX4. This basic
data receipt concept is shown in Exhibit 5.




                                  Exhibit 5. AWIPS II Data Receipt




Lecture 1                                      - 10 -                AWIPS II System Architecture
AWIPS II SA Training, Release 2                                        NWSTC Training Guide


As shown in Exhibit 5:
        1. The AWIPS II interface process obtains a data product from the data source.
        2. The interface process writes the data to a file in Raw Data Storage.
        3. The interface process posts a “data available” message to the QPID message
           broker.
        4. The EDEX Ingest process obtains the “data available” message from QPID
           and removes the message from the message queue.
        5. The EDEX Ingest process obtains the data file from Raw Data Storage.
AWIPS II either modifies or provides interfaces between the existing AWIPS
components and AWIPS II data ingest. The AWIPS II interfaces are shown in Table 6.

Table 6. AWIPS II Interfaces to Data Sources
   Source                                 AWIPS II Interface
SBN             LDM(U) on CPSBN1/2 cluster obtains data from the SBN and feeds the
                data to LDM(D) on the DX1/2 cluster. LDM(D) writes the data to a file in
                Raw Data Storage and posts a “data available” message to the QPID
                message broker on the CPSBN1/2 cluster.
LDAD            LDAD has been modified to feed data into the AWIPS II ingest data flow.
ORPG/SPG The Radar Server on DX1/2 cluster interacts with ORPG/SPG to obtain
         radar data. When the data is obtained, the server writes the data to a file in
         Raw Data Storage and posts a “data available” message to the QPID
         message broker on the CPSBN1/2 cluster.

Note that this architecture provides separation between data sources and ingest
processing. Any data source that follows this architecture will be able to provide data for
EDEX to process.

Basic AWIPS II Data Decoding Architecture
Data decoding is defined as the process of converting data from a raw format into a
decoded format that is usable by the AWIPS II visualization software. In AWIPS II, data
decoding is performed by the EDEX Ingest processes, which run on both DX3 and DX4.
The basic data decoding concept is shown in Exhibit 6.




Lecture 1                                  - 11 -                AWIPS II System Architecture
AWIPS II SA Training, Release 2                                        NWSTC Training Guide




                              Exhibit 6. AWIPS II Data Decoding

As shown in Exhibit 6:
        1. The EDEX Ingest process obtains the “data available” message from the
           QPID message broker. The EDEX Ingest process determines the appropriate
           data decoder based on the message contents and forwards the message to the
           decoder. Finally, the message is removed from the message queue.
        2. The EDEX Ingest process reads the data from Raw Data Storage.
        3. The EDEX Ingest Process decodes the data.
        4. The EDEX Ingest process forwards the decoded data to Data Storage.
        5. The EDEX Ingest process sends a message to a client queue that decoded data
           is available.

The current architecture has two EDEX Ingest processors, DX3 and DX4. Load
balancing of Ingest processing among multiple Ingest processes is provided by the QPID
message broker. Although multiple EDEX Ingest processes are possible, each “data
available” message is serviced by whichever EDEX Ingest processor gets the message
first. Either EDEX Ingest processor may be taken offline without stopping data ingest.

It is also important to note that in AWIPS II all data types are processed on both Ingest
Processors. Although a specific data file is processed by a single EDEX Ingest process,
data files for a data type are processed on both Ingest processors.

Once this process is complete, clients may obtain and perform additional processing on
the data. This includes display of the data in CAVE, additional processing such as Smart
Init scripts that are run on Gridded Binary (GRIB) data, text watch/warn triggered scripts,
etc.




Lecture 1                                   - 12 -                AWIPS II System Architecture
AWIPS II SA Training, Release 2                                         NWSTC Training Guide



Basic AWIPS II Processed Data Storage Architecture
Processed Data Storage refers to the persistence of decoded data and occurs in two
separate forms: 1) metadata and some decoded data, which is stored in PostgreSQL
database tables; and 2) imagery data, gridded forecast data, and certain point data, which
is stored in Hierarchical Data Format – 5 (HDF5) files. HDF5 storage is managed by the
Python Process Isolated Enhanced Storage (PyPIES) server. The basic data storage
concept is shown in Exhibit 7.



        EDEX                 1                              2         HDF5 Data
                                      PyPIES
       Process                                                          Store


                   3                                        4          Metadata
                                        DBMS
                                                                       Database

                         Exhibit 7. AWIPS II Processed Data Storage

As shown in Exhibit 7, writing to the Processed Data Store involves the following:
        1. The EDEX Process sends serialized data, area data, and certain point data to
           the PyPIES Process.
        2. The PyPIES Process writes the data to the HDF5 Data Store.
        3. The EDEX Process sends the metadata to the PostgreSQL DBMS.
        4. The PostgreSQL DBMS writes the metadata to the AWIPS II database.
Note that not all data is stored in HDF5. For data not stored in HDF5, Steps 1 and 2 are
skipped.

For data retrieval, the process is reversed:
        1. The EDEX Process requests the metadata from the PostgreSQL DBMS.
        2. The PostgreSQL DBMS reads the metadata from the AWIPS II database and
           returns it to the EDEX Process.
        3. The EDEX Process sends a data request to the PyPIES Process.
        4. The PyPIES Process reads the data from the HDF5 Data Store and returns it to
           the EDEX Process.
In this case, if the data is not stored in HDF5, then Steps 3 and 4 are skipped.

This architecture provides a separation of the EDEX processes from the physical data,
increasing the reliability of the system by decoupling the physical data from the EDEX


Lecture 1                                      - 13 -             AWIPS II System Architecture
AWIPS II SA Training, Release 2                                           NWSTC Training Guide


processes and allowing Processed Data Storage to reside on separate servers from the
ingest processing. Note that data persistence to the HDF5 Data Store is performed before
client notification takes place; this ensures that clients only attempt to access data that has
been stored.

Python Process Isolated Exchange Storage (PyPIES)
PyPIES was created for AWPS II to isolate the management of HDF5 Processed Data
Storage from the EDEX processes. PyPIES manages access, i.e. reads and writes, of data
in the HDF5 files. In a sense, PyPIES provides functionality similar to a DBMS; all data
being written to an HDF5 is sent to PyPIES, requests for data from an HDF5 file are
processed by PyPIES.

PyPIES is implemented in two parts:
        1. The PyPIES manager is a Python application that runs as part of an Apache
           HTTP server on DX1F. It is the PyPIES manager that handles requests to
           store and retrieve data. The Apache HTTP server provides request
           management and isolates the HDF5 read/write to a separate process.
        2. The PyPIES logger is a Python process that coordinates logging.

PyPIES is controlled by the httpd-pypies System 5 script; it is part of the a2dx1apps high-
availability package.

Basic AWIPS II Data Retrieval Architecture
Data retrieval is the process used by an AWIPS II client to obtain data with the EDEX
Servers. In making the data retrieval, the client interacts with an EDEX Request process;
the Request process obtains the requested data from the Processed Data Store and returns
it to the client. The basic data retrieval process is shown in Exhibit 8.



                            1
      CLIENT                            IPVS

                                             2


                       4               EDEX                  3              Data
                                      Request                              Storage
                                Exhibit 8. AWIPS II Data Retrieval




Lecture 1                                     - 14 -                 AWIPS II System Architecture
AWIPS II SA Training, Release 2                                        NWSTC Training Guide


As shown in Exhibit 8:
        1. The client sends a request via TCP protocol to a virtual host name managed by
           IPVS.
        2. IPVS forwards the request, which includes the return address of the client, to
           an available EDEX Request process.
        3. The EDEX Request process obtains the requested data from the AWIPS II
           Processed Data Store.
        4. The EDEX Request process forwards the requested data directly to the client.
This architecture allows the client access via any available EDEX Request process
resulting in improved system reliability and accessibility. Note that data retrieval from
the Processed Data Store follows the same pattern as data storage: retrieval of HDF5
stored data is performed via the PyPIES process; retrieval of database data is via the
PostgreSQL DBMS.

BASIC AWIPS II DATA RETRANSMISSION DESIGN
AWIPS II uses the sequence numbers provided by the SBN to implement an automated
retransmission capability for missed products delivered by the SBN. The retransmission
capability has been implemented via a daemon process, sbn_retransmit, which is installed
as part of the LDM installation on the CPSBN1/2 cluster. The AWIPS II Message
Retransmission Design is shown in Exhibit 9. Notice that AWIPS II reuses most of the
existing retransmission infrastructure.




                        Exhibit 9. AWIPS II Message Retransmission


Lecture 1                                   - 15 -                AWIPS II System Architecture
AWIPS II SA Training, Release 2                                          NWSTC Training Guide


To determine if a product retransmission is required, sbn_retransmit monitors the LDM
product queue. When a gap is detected, sbn_retransmit forwards a request to the Network
Control Facility (NCF) for a retransmission of the missing products.

BASIC AWIPS II DATA PURGE ARCHITECTURE
AWIPS II stores data in two separate collections: 1) the AWIPS II Raw Data Storage,
which contains unprocessed data; and 2) the AWIPS II Processed Data Storage, which
contains decoded and other processed data. These two data collections use different purge
mechanisms.

AWIPS II Processed Data Storage Purge Architecture
AWIPS II has implemented a data plug-in based purge strategy. The default purge
strategy is version-based, similar to AWIPS I’s version-based purge strategy. Each data
plug-in is able either to use the default strategy or to implement its own strategy. The
GFE and text data plug-ins implement their own purge strategies based on the purge
strategies used in AWIPS I; all other data plug-ins currently use the default, version-
based purge strategy.

Rules for the version-based purge are defined in XML files located in the AWIPS II
Localization Store, which is in /awips2/edex/data/utility on DX3/4. See SMM Chapter 11
for more information on the purging and purge rules.

The basic Processed Data Storage purge strategy is shown in Exhibit 10.



                         EDEX            2                      3           Metadata
      QUARTZ                                      DBMS
                         Ingest                                             Database
            1


                                     4                          5          HDF5 Data
                                                 PyPIES
                                                                             Store

                       Exhibit 10. AWIPS II Processed Data Storage Purge

As shown in Exhibit 10:
        1. A quartz event is triggered in the EDEX Ingest process causing the Purge
           Service to execute.
                a. The EDEX Purge Service obtains a purge lock; if the lock is already taken,
                   the Purge Service exits. This strategy ensures that only a single EDEX
                   Ingest process actually performs the purge.
                b. The EDEX Purge Service obtains the purge rules for each data plug-in.
        2. The EDEX Purge Service sends a delete message to the PostgreSQL DBMS.


Lecture 1                                     - 16 -                AWIPS II System Architecture
AWIPS II SA Training, Release 2                                        NWSTC Training Guide


        3. The PostgreSQL DBMS deletes the data from the database table(s).
        4. If HDF5 data is to be purged, the EDEX Purge Service sends a purge message
           to the PyPIES service.
        5. The PyPIES service deletes the data from the HDF5 file(s).

AWIPS II Raw Data Storage Purge Architecture
Purging the AWIPS II Raw Data Storage is managed by a cron job on the LDM(D) server
(dx2f). This cron causes the LDM scour process to run. The purge process is age based.
The directories to manage, and the retention time for each directory, are configured in the
scour.conf file located in the etc directory under the LDM home directory (normally
/usr/local/ldm). The AWIPS II Raw Data Storage purge process is shown in Exhibit 11.



                                     2                        3          AWIPS II Data
       CRON         ldmadmin                        scour
            1                                                              Archive

                        Exhibit 11. AWIPS II Raw Data Storage Purge

As shown in Exhibit 11:
        1. An ldm user cron job executes ldmadmin.
        2. ldmadmin executes the LDM scour program.
        3. The LDM scour program deletes outdated data from AWIPS II Raw Data
           Storage.




Lecture 1                                  - 17 -                 AWIPS II System Architecture
AWIPS II SA Training, Release 2                                        NWSTC Training Guide




REVIEW QUESTIONS
    1. Which of the following are components of AWIPS II?
            a. Rehosted AWIPS I applications
            b. Reengineered AWIPS I applications
            c. Wrapped AWIPS I applications
            d. Unchanged AWIPS I applications
            e. All of the above

    2. Linux High-Availability server runs on DX3/4 to failover the EDEX cluster.
                TRUE                    FALSE

    3. Match the following servers/workstations with the appropriate software installed
       for AWIPS II (software is installed but not necessarily running if tied to a failover
       package).
                1. CP 1/2         __________            a. CAVE

                2. LX             __________            b. AlertViz

                3. XT             __________            c. LDM

                4. DX1/2          __________            d. QPID

                5. DX3/4          __________            e. IPVS

                6. PX1/2          __________            f. LDAD

                                                        g. Linux Heartbeat

                                                        h. PostgreSQL

                                                        i. Radar Server

                                                        j. PyPIES

                                                        k. MHS

                                                        l. EDEX




Lecture 1                                      - 18 -             AWIPS II System Architecture
AWIPS II SA Training, Release 2                                         NWSTC Training Guide


    4. AWIPS II uses floating server names to connect to servers running high-
       availability packages. Match the floating server name on the left with the
       associated AWIPS II high-availability package on the right.
            1. dx1f __________                   a. a2px1apps

            2. dx2f __________                   b. a2dx1apps

            3. px1f __________                   c. a2cp1apps

            4. px2f __________                   d. a2px2apps

            5. cp1f __________                   e. a2dx2apps

                                                 f. a2cp2apps


    4. What is the purpose of the ec (aka edexcluster) server alias?
              a. It is an alias for dx3f, which is used for failover management of the DX3
                 and DX4 EDEX Servers.

              b. It points to CPSBN1 or CPSBN2 where IPVS redirects connections to
                 DX3 and DX4 for load balancing. It applies only to connections on port
                 9581.

    5. DX3/4 are not a cluster pair; they are load balanced. If one server fails, the other
       server will manage all data processing responsibilities.

                 TRUE                    FALSE

    6. Match the data stored on the DAS with the server(s) that can access the data.

                 dx2f             dx1f         dx3/4            various systems

              a. /aiildm is accessed from _____________

              b. /aiihdf5 is accessed from _____________

              c. /aiidb is accessed from _____________

    7. Match the data stored on the NAS with the server(s) that can access the data.

                 dx2f             dx1f         dx3/4            various systems

              a. /aiidata is accessed from _____________
              b. /aiidata/manual is accessed from _____________
              c. /aiidata/hdf5 is accessed from _____________



Lecture 1                                    - 19 -                AWIPS II System Architecture
AWIPS II SA Training, Release 2                                         NWSTC Training Guide


              d. /GFESuite2 is accessed from _____________


    8. Match the data flow processes on the left with the definitions on the right. There
       is only one answer per match.

            a. Visualization      ___   Data product received from LDAD, SBN, or ORPG
                                        and written to Raw Data Storage  “data available”
                                        message sent to QPID message broker EDEX
                                        Ingest process receives “data available” message
                                        from QPID  EDEX Ingest process obtains the data
                                        from AWIPS II Raw Data.
            b. Data Storage       ___   EDEX Ingest process determines the appropriate
               Purging                  data decoder for the data EDEX Ingest process
                                        forwards the message to the decoder and removes
                                        the message from the message queueEDEX Ingest
                                        decodes the dataEDEX Ingest process forwards
                                        the decoded data to data storage and notifies the
                                        client queue that decoded data is available.
            c. Data Storage       ___   EDEX Process sends data to PyPIES PyPIES
                                        writes the data to the HDF5 Data StoreEDEX
                                        Process sends the metadata to the PostgresSQL
                                        DBMSPostgresSQL DBMS writes the metadata to
                                        the AWIPS II database.
            d. Data Decoding      ___   CAVE data request sent to IPVS IPVS forwards
                                        data request to EDEX EDEX Request process
                                        sends requested data to caveCAVE displays data
                                        for user.
            e. Data Retrieval     ___   EDEX Process requests the metadata from the
                                        PostgresSQL DBMS PostgresSQL DBMS re turns
                                        the metadata to the EDEX Process Edex Process
                                        sends a data request the PyPIESPyPIES reads the
                                        data from the HDF5 Data Store and sends it to the
                                        EDEX Process.
            f. Data Receipt       ___   A quartz event is triggered in the EDEX Ingest
                                        process causing the Purge Service to execute
                                        EDEX Purge Service sends a delete message to the
                                        PostgresSQL DBMSPostgresSQL DBMS deletes
                                        the data from the database table(s) If HDF5 data is
                                        to be purged a purge message is sent to PyPIES
                                        PyPIES deletes the data from the HDF5 Data Store.




Lecture 1                                     - 20 -              AWIPS II System Architecture
AWIPS II SA Training, Release 2                                         NWSTC Training Guide




OBJECTIVES REVIEW
   Upon completion of this module, the ESA/ITO will understand the overall AWIPS II
    architecture.
        AWIPS II represents a partial replacement of the existing functionality, primarily
        the data processing and forecaster tools. It has been designed to improve system
        reliability through the use of distributed processing. It also provides interfaces to
        existing functionality, e.g., the LDAD, which is not being replaced.

   Upon completion of this module, the ESA/ITO will know where the AWIPS II
    software is deployed.
        AWIPS II installs new software on several existing servers. These include
        CPSBN1 and CPSBN2, PX1 and PX2, DX1 and DX2, and DX3 and DX4.
        AWIPS II software is also installed on the LX and XT workstations. AWIPS II
        also rehosts existing software on selected servers and interfaces with existing
        software systems such as the SBN and LDAD.

   Upon completion of this module, the ESA/ITO will have the knowledge to describe
    the patterns used in storage of raw (Raw Data Storage) and processed (Processed
    Data Storage) data in the AWIPS II system.
        For AWIPS II, an application inserting data into the Data Ingest writes data to
        Raw Data Storage and sends a message to QPID to initiate processing. The
        Unidata LDM software acts as the bridge between the SBN and the AWIPS II
        system. This interface includes an auto reship capability and Raw Data Storage
        purging. The LDAD has been modified to interface with the AWIPS II system.
        AWIPS II introduces a new process, the Radar Server, to provide the interface
        between the ORPG and the AWIPS II systems.




Lecture 1                                   - 21 -                 AWIPS II System Architecture


				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:65
posted:3/5/2012
language:Latin
pages:25