; ML_march041
Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out
Your Federal Quarterly Tax Payments are due April 15th Get Help Now >>

ML_march041

VIEWS: 5 PAGES: 24

  • pg 1
									          Iosif Legrand

California Institute of Technology
               March 2004            Iosif Legrand
              Distributed Dynamic Services
                       Architecture
   MonALISA is a Distributed, Dynamic Service Framework


   Hierarchical structure of loosely coupled services which are
    independent & autonomous entities able to cooperate using a
    dynamic set of proxies or self describing protocols.


   They need a dynamic registration and discovery & subscription
    mechanism. For an effective use of distributed resources, these
    services should provide adaptability and self-organization
    (aggregation and hierarchical orchestration)


   Flexible access to complex monitoring information.



                                March 2004               Iosif Legrand
                     Distributed Object Systems
                        CORBA , RMI, DCOM



“Traditional” Distributed Object Models                The Stub is linked to the Client.
           (CORBA, DCOM)                               The Client must know about the
                                                       service from the beginning and
                                                       needs the right stub for it

                       Lookup
                      Skeleton                             Lookup
                                                           Stub
           Server      Service                             Service
                                                                      CLIENT


                                                                     “IDL” Compiler




          The Server and the client code must be created together !!

                                          March 2004                    Iosif Legrand
          Web Services WSDL/SOAP




                      SOAP

Server                                         CLIENT
Lookup
WSDL                                            Lookup
                                              Interface
Service                                         Service




The client can dynamically generate the data structures and
the interfaces for using remote objects based on WSDL


               Platform independent


                             March 2004            Iosif Legrand
                    Distributed Services
            Any well suited protocol for the application
                                                    Lookup
                                                    Proxy
 Service                                            Service
                                                              CLIENT
 Lookup
 Proxy
  Service
                Dynamic Code Loading

            Services can be used dynamically
    Remote Services                   Proxy == RMI Stub
    Mobile Agents                     Proxy == Entire Service
    “Smart Proxies”                   Proxy adjusts to the client
Act as a true dynamic service and provide the necessary
functionally to be used by any other services that require such
information (Jini, interface to WSDL / SOAP)
    mechanism to dynamically discover all the “Service Units"
    remote event notification for changes in the any system
    lease mechanism for each registered unit
                                      March 2004                 Iosif Legrand
                                        JINI – Network Services

              jar
                                                                   A Service Registers with at least one
                    Web
                                                                   Lookup Service using the same ID.
                    Server
    Publish the                                                    It provides information about its
                                           CLIENT
    “Interface” jar                    S                           functionality and the URL addressed
                                           P
                                                                   from where interested clients may
                Lookup
                Lookup
                Service                                            get the dynamic code to use it.
                Register                                           The Service must ask each Lookup
                                                                   Service for a lease and periodically
                                                                   renew it.
Service ID                   Register with ID
                                                    Lookup
                                                    Service
             Servic
                                                                   If a Service fails to renew the lease, it
               e                                                   is removed form the Lookup Service
                                Ask for a lease                    Directory. When problems are
                               Get a lease for DT                  solved, it can re-register.
             The lease mechanism allows the Lookup Service to keep an up to
             date directory of services and correctly handle network problems.

                                                              March 2004                  Iosif Legrand
  Monitoring: Data Collection

                     Dynamic
          PULL                          Other tools
                    Thread Pool         (Ganglia, MRT…)

    SNMP get & walk
 rsh | ssh remote scripts                 Farm            Configuration Control
End-To-End measurements                  Monitor

                                                    Trap
                                                  Listener


                       PUSH
                                                     WEB Server
                     snmp trap
                                                     Dynamic loading of
                                                     modules or agents


                               Trap Agent
                            (ucd – snmp) perl


                        March 2004                           Iosif Legrand
                   The Muti-Threaded Execution
                           Architecture




•Each request is done in an independent thread

•A slow agent / busy node does not perturb the measurements of an entire system

•Ex: Monitor 300 nodes @ 30 seconds interval 10-15 Threads are running in parallel
                                        March 2004                Iosif Legrand
                                Data Handling

Data Model                                                             TIME
   Configuration      Farm , Function (Cluster), Node, Module
   Monitored Values                                             {Parameter, Value}


   (Automatic) Mapping of the Data Model in :
        XML, SQL, SOAP, …


   Configuration & Results objects are store in a DB
        (dynamically configurable for InstanDB, Postgres, MySQl, Oracle … )
   Subscription to results objects matching a template / predicate
   Clients can load filter objects into the Data Cache service and generate any
    derived (or aggregate) data structures and register to receive them.
   Monitored parameters have a life time
                                            March 2004               Iosif Legrand
                  Service Monitor UNIT & Data Handling

                                         Monitor                            Lookup     Lookup
                           WEB          Data Stores                         Service    Service
    Client                Service
(other service)
  Web client              WSDL        Data Cache
                          SOAP       Service & DB



                                                                     data
                        McKoi DB
                        MySQL                                                                        Client
                                                                 Predicates & Agents             (other service)
                                      Farm                                                            Java
          Other tools                 Monitor           Configuration Control (SSL)

                                                         UDP
                                                                    MySQL
                                                                               Predicates & Agents
                                     User defined loadable
                                    Modules to write /sent data        MDS

                                                    March 2004                        Iosif Legrand
      Registration / Discovery / Remote
                 Notification

MonALISA    Registration                   Discovery
 Service
                           Lookup                                     Client
                           Service                                (other service)

                                             Services
                                              Proxy
                                            Multiplexer

MonALISA
 Service                                                         Data
                                            Services
                                                          Filters & Agents
                                             Proxy
                                           Multiplexer

                           Lookup
 MonALISA                  Service
  Service                                                              Client
                                                                   (other service)



                              March 2004                   Iosif Legrand
         Secure – Automatic Update Mechanism
     for Services, Clients & Embedded Applications
                         Lookup
                         Service
Key store
            MonaLisa   Discovery                       Web Server
             Service    download
                                                    Sign Distribution

                                                                   All running services
                                                                    are update using the
                                                                    discovery mechanism
                                            Admin Client
                                                                   At startup each service
                                                   Priv. Key        check if it an update is
Key store                                                           done at a set of Web
            MonaLisa                                                Servers
             Service        Lookup
                            Service
                                                                   Clients use the Web
                                                                    Start mechanism

                       EMBEDDED APPLICATIONS
                       Monitor, Control Execution, Update

                                      March 2004                         Iosif Legrand
        Global Client for Farms and
           Network Connectivity

@ CALTECH




                                     DataTAG
                                 -
                            -




                    March 2004         Iosif Legrand
Real-time Data for Large Systems
“lxshare” cluster at cern ~ 600 ndoes




                 March 2004         Iosif Legrand
Access to historical and real-time values
                                Past values are presented and the GUI
                                remains a registered listener and
                                the new vales are added




   Real Time Histograms for various parameters




                               March 2004                   Iosif Legrand
                 Mobile Agents and Filters



                                                    Simple “Global Load” filter agent




                    From FNAL to all



Maximum Flow Data Replication
Path Agent Deployed to each RC
and evaluates the best path for                                     From CERN to all
real-time data replication
                                       March 2004                    Iosif Legrand
  Pseudo – Clients & Dedicated Repositories

                    Lookup
                    Service
MySQL
 IDB    MonaLisa   Discovery
         Service
                                                           WAP


                                             TOMCAT
                                            JSP/servelts
                                                                 WEB
                                     Pseudo Client

                                            MySQL
MySQL
 IDB    MonaLisa
         Service

                     Lookup
                     Service




                               March 2004                        Iosif Legrand
MonALISA repositories




         March 2004     Iosif Legrand
                              VRVS Architecture
                                                            pub
cal-            cor-
tech            nell

                                                  funet
                                                                  Reflectors are hosts that
vrvs            star-          vrvs                               interconnect users by
  5             light           us                                permanent IP tunnels.
                                                 vrvs
                                                  eu

                                                                  The active IP tunnels must
                usf                                               be selected so that there is
                                                                  no cycle formed.

                                      sinica
        inet
          2
                        usp                                                Tree

                                                                  The selection is made
                                      kek                         according to the assumed
       triumf
                                                                  network links performance.

                                               March 2004                    Iosif Legrand
         Minimum Spanning Tree Algorithms

Finding a tree T that contains all the vertices
of a graph G        spanning tree
and has the least total weight over all
such trees        minimum-spanning tree (MST)

 Input: A weighted connected graph G = (V,E) with n vertices
        and m edges

 Output: A minimum- spanning tree T
                   w(T )       w((v, u))
                             ( v ,u )T

                                          March 2004   Iosif Legrand
Monitoring VRVS Reflectors




            March 2004       Iosif Legrand
Monitoring VRVS Reflectors




           March 2004        Iosif Legrand
Global Client / Dynamic Discovery




               March 2004     Iosif Legrand
                                   SUMMARY
   MonALISA is able to dynamically discover all the “Service Units" used by a
    community and through the remote event notification mechanism keeps an
    update state for the entire system
   Automatic & secure code update (services, embedded applications and clients) .
   Dynamic configuration for services. Secure Admin interface.
   Access to aggregate farm values and all the details for each node
   Selected real time / historical data for any subscribed listeners
   Active filter agents to process the data and provided dedicated / customized
    information to other services or clients.
   Mobile Agents for decision support and global optimization.
   Dynamic proxies and WSDL & WAP pages for services.
   Embedded SNMP support and interfaces with other tools ( LSF, PBS, Ganglia,
    Hawkeye, IEPM-BW, MRTG, RRD, Generic Web Services…)
   Dedicated pseudo-clients for repository, WAP access or decision making units
   It proved to be a stable and reliable distributed service system. It is currently
    running at ~ 120 sites
                   http://monalisa.cacr.caltech.edu
                                          March 2004                Iosif Legrand

								
To top