Information and Monitoring Status and Plans

Document Sample
Information and Monitoring Status and Plans Powered By Docstoc
					                                       Enabling Grids for E-sciencE

                    Building a robust distributed
                      system: some lessons
                            from R-GMA
                  CHEP-07, Victoria, Canada, 3-7 September 2007

                  Steve Fisher/RAL on behalf of JRA1-UK

                      Enabling Grids for E-sciencE

    • GMA and R-GMA overview
    • Managing Memory Usage
    • Buffers
         – Consumer
         – Primary producer
         – Secondary producer
    • Loss of control messages
    • Replication
         – Schema
         – Registry
    • Conclusions

INFSO-RI-508833                                      Steve Fisher/RAL   2
                     Enabling Grids for E-sciencE

  • Defined by the GGF                                Producer
        – Now OGF
  • 3 Components


        – Producer
        – Consumer
        – Registry                                    Consumer

  • Real system needs to tie                        • The INFOD-WG at GGF
    down message formats                               – IBM, Oracle and others have
        – This has been done by                          defined a GMA compliant
          R-GMA                                          specification

INFSO-RI-508833                                                        Steve Fisher/RAL   3
                    Enabling Grids for E-sciencE

    • Relational implementation of                 Producer
      the GGF’s GMA
    • Provides a uniform method to

      publish and access both                                         Registry

      information and monitoring
    • Registry is hidden                           Consumer

    • It is intended for use by:
         – Other middleware
         – End users
    • Easy for individuals to define,
      publish and retrieve data
    • All data has a timestamp,
      enabling its use for

INFSO-RI-508833                                                   Steve Fisher/RAL   4
                                                           R-GMA Producers
                       Enabling Grids for E-sciencE

    • Primary – source of data
    • Secondary – republish data
         – Co-locate information to speed up queries
         – Reduce network traffic




                                                             PP – Primary Producer
        PP                                                   SP – Secondary Producer

INFSO-RI-508833                                                        Steve Fisher/RAL   5
                                  Three points in the R-GMA evolution
                          Enabling Grids for E-sciencE

    • EDG
         – corresponding to the version developed within EDG.
    • EGEE-I
         – for the version deployed in gLite 3.0
    • EGEE-II
         – for the version that will be rolled out late Summer and Autumn of
           2007 as upgrades to gLite 3.1
                   Designed to address properly all the long term problems

INFSO-RI-508833                                                        Steve Fisher/RAL   6
                                                        Managing Memory Usage
                         Enabling Grids for E-sciencE

    • R-GMA may have varying amounts of memory available
         – May share servlet container (Tomcat) with other servlets
         – JVM may be badly configured
    • EGEE-II solution
         – Use JDK 5 Observer to monitor memory usage
         – When memory low RGMABusyException returned for all user
           calls that may take extra memory
                   inserting data into the system
                   creating new producer or consumer resources
    • We try to be fair
         – If you behave reasonably you should not be penalised
         – If problem is caused by too many reasonable demands must
           reject requests with the RGMABusyException.

INFSO-RI-508833                                                      Steve Fisher/RAL   7
                                  Avoiding bottlenecks in the data flow
                   Enabling Grids for E-sciencE

                     Primary                      Secondary
                     Producer        B             Producer B
                      Service                       Service

                                                                Consumer B            Consumer
                                                                 Service                Client

                     Primary                      Secondary
                     Producer        B             Producer B
                      Service                       Service

    • Buffers are shown “B”
         – Primary producer
         – Secondary producer
         – Consumer

INFSO-RI-508833                                                              Steve Fisher/RAL    8
                                 Consumer Buffering - problem
                    Enabling Grids for E-sciencE

    • Consumer has a buffer for each client where the results
      of the query are stored until they are popped
         – If the application is slow to pop() then buffers can fill up.
         – Cannot send an RGMABusyException to a pop() as this call
           reduces memory use.

INFSO-RI-508833                                               Steve Fisher/RAL   9
                                  Consumer Buffering - solution
                    Enabling Grids for E-sciencE

    • EGEE-I solution
         – Allocate each instance a certain amount of memory
         – When this is full data are written to disk
         – Once the data are all read from disk, the disk file is removed and
           memory is used again.
         – If allocation on disk runs out we close the consumer.
         – This allows us to cope with peaks of data and is working well

INFSO-RI-508833                                                Steve Fisher/RAL   10
                                                 Primary Producer Buffering
                          Enabling Grids for E-sciencE

    • Problem only exists with memory storage
         – Latest store to answer latest queries
                   Must hold tuples up to their LRT
         – History store for history and continuous queries
                   Must hold enough tuples to satisfy the history retention period
                   Must hold tuples for which delivery to existing continuous queries
                    has not been attempted.

INFSO-RI-508833                                                          Steve Fisher/RAL   11
                        Primary Producer Buffering - solution
                   Enabling Grids for E-sciencE

    • EGEE-I solution
         – RGMABufferFullException which is thrown when a producer tries
           to publish a new tuple and the producer has exceeded a server
           defined limit.
         – Works most of the time
    • EGEE-II extra
         – We have the RGMABusyException to fall back on.

INFSO-RI-508833                                             Steve Fisher/RAL   12
                                 Secondary Producer Buffering
                    Enabling Grids for E-sciencE

    • In EDG and EGEE-I designs the secondary producer
      was made up of consumers and one producer.
    • In the EGEE-II design incoming data are stored directly
      in the tuple store.
    • A memory based tuple store can grow very large but
      nobody to send an RGMABusyException to.
    • For this reason we do not generally recommend using
      memory based tuple stores for secondary producers.
    • Will close the secondary producer when unable to deal
      with memory demands implied by retention periods.
         – This will be added to the servlet code that would normally be
           sending the RGMABusyException.

INFSO-RI-508833                                               Steve Fisher/RAL   13
                                  Coping with loss of control messages
                              Enabling Grids for E-sciencE

                                                             Consumer            • EDG
    Service                                                   Service
                                                                                       – Register once
                   Refresh               Register
                                                                                       – Refresh periodically
       Register                                      Refresh            Notification   – Only register results in
                                                                                         notification and start
   EDG                       Registry
                                                                                       – Network problems can
                                                                                         block everything
     Service                   Data
                                                              Service            • EGEE-I
                                                                                       – Use register as refresh
                                      Register/Refresh                                      No longer need
                                                                        Notification         messages to get
                                             Q                                              But much more traffic
                                             Q                                         – Split queue into slow
                                             Q                                           medium and fast queue

INFSO-RI-508833                                                                                      Steve Fisher/RAL   14
                                  Coping with loss of control messages
                          Enabling Grids for E-sciencE

   Producer                                          Consumer
    Service               Data                        Service


                         Registry       Q
    EGEE-I               Service
                                        Q                          • EGEE-II
                                                                            – For a producer a register
   Producer                Start                    Consumer
                                                                              message now return the
    Service               Data                       Service                  consumers of interest and vice
                                                                            – Producers now notify
      Register/Refresh                        Register/Refresh
                                                                              consumers themselves
                                                                            – Messages to other servers go
   EGEE-II               Registry                                             via a task on the task queue

INFSO-RI-508833                                                                               Steve Fisher/RAL   15
                                                     Task Manager
                    Enabling Grids for E-sciencE

    • Assumption is that tasks dependent upon some
      unreliable resource
         – e.g. network connection to a server and that server
    • Assign a key to each resource
    • One queue of tasks but a pool of task invocators
    • Initially empty set of good keys
    • If a task is successful on its first try its key is added to
      good set
    • If a task fails its key is removed from good set
    • Only if key is in good set will more than one task be run
      with that key
    • Some invocators only take tasks with a good key

INFSO-RI-508833                                              Steve Fisher/RAL   16
                                                               Schema Replication
                        Enabling Grids for E-sciencE

               Server            2
                                                           • VDB is defined by a
                   4                                         configuration file
                                 U                           U
                                                             identifying the master
                                                           • Each server has full
                                          Server             information for each VDB
                                                           • Tried to avoid a master but
                                                             very difficult
                                                           • Updates are first done on
                                                             the master
                                                           • Replication request is
                                                             “updates since”

INFSO-RI-508833                                                              Steve Fisher/RAL   17
                                                    Registry replication
                     Enabling Grids for E-sciencE

    • A registry “owns” those records that were last changed by direct
      calls and is responsible for pushing updates of these records.
    • The registry is updated:
         – by direct calls – sets master flag
         – upon receipt of replication messages – clear master flag
    • If registry unavailable direct update requests are routed to a
      different registry instance and records in the new registry will get
      the master flag set.
         – The system will clean up when a registry assumes mastership for the
           record and replicates its records.
    • A hash table is used to hold the add registration and delete
      registration requests keyed on the primary key of the entry.
         – Each replication cycle a new hash table is created to take new entries
           and the old one is processed.
    • Time stamps are associated with the replication messages
         – Can recognise missed messages and recover

INFSO-RI-508833                                                    Steve Fisher/RAL   18
                    Enabling Grids for E-sciencE

    • Try to think of everything that can go wrong.
    • Keep it simple.
    • Polling is much simpler though less efficient than notification.
    • Make the system self correcting and avoid critical messages.
    • Avoid single points of failure.
    • Reject incoming requests if a server cannot cope rather than just going
      slowly or crashing.
    • Server code should protect itself against running out of memory.
    • External conditions can change at any time: it is not good enough to just
      check at service startup.

                    This will give us a highly
                    robust and scalable R-GMA.

INFSO-RI-508833                                                 Steve Fisher/RAL   19

Shared By: