The Emergence of Open Grid Standards

Document Sample
The Emergence of Open Grid Standards Powered By Docstoc
					                                         The Emergence of
                                        Open Grid Standards

                                                                                 Managed shared
                                        Computer science research
Increased functionality,




                                                                                  virtual systems
    standardization




                                                                  Open Grid
                                  Web services, etc.
                                                                 Services Arch
                                                                Real standards
                                                           Multiple implementations
                            Internet
                                          Globus Toolkit
                           standards
                                           Defacto standard
                             Custom      Single implementation
                            solutions

                           1990           1995             2000           2005               2010
Open Grid Services Architecture                                                                     1
         Open Grid Services Architecture
        Service orientation to virtualize resources
            Everything is a service
        From Web services
            Standard interface definition mechanisms
            Evolving set of other standards: security, etc.
        From Grids (Globus Toolkit)
            Service semantics, reliability & security models
            Lifecycle management, discovery, other services
        A framework for the definition & management
         of composable, interoperable services
    “The Physiology of the Grid: An Open Grid Services Architecture for
  Distributed Systems Integration”, Foster, Kesselman, Nick, Tuecke, 2002
Open Grid Services Architecture                                             2
        The Open Grid Services Architecture




Open Grid Services Architecture               3
                            Web Services
         XML-based distributed computing technology
         Web service = a server process that exposes
          typed ports to the network
         Described by the Web Services Description
          Language, an XML document that contains
             Type of message(s) the service understands &
              types of responses & exceptions it returns
             “Methods” bound together as “port types”
             Port types bound to protocols as “ports”
         A WSDL document completely defines a
          service and how to access it
Open Grid Services Architecture                              4
                          WSDL Example
       <wsdl:definitions targetNamespace=“…”>
         <wsdl:types>
           <schema>
             <xsd:element name=“fooInput” …/>
             <xsd:element name=“fooOutput” …/>
           </schema>
         </wsdl:types>
         <wsdl:message name=“fooInputMessage”>
           <part name=“parameters” element=“fooInput”/>
         </wsdl:message>
         <wsdl:message name=“fooOutputMessage”>
           <part name=“parameters” element=“fooOutput”/>
         </wsdl:message>
         <wsdl:portType name=“fooInterface”>
           <wsdl:operation name=“foo”>
             <input message=“fooInput”/>
             <output message = “fooOutput”/>
           </wsdl:operation>
         </wsdl:portType>
       </wsdl:definitions>

Open Grid Services Architecture                            5
              Transient Service Instances
         “Web services” address discovery & invocation
          of persistent services
             Interface to persistent state of entire enterprise
         In Grids, must also support transient services,
          created/destroyed dynamically
             Interfaces to the states of distributed activities
             E.g. workflow, video conf., dist. data analysis
         Significant implications for how services are
          managed, named, discovered, and used
             In fact, much of our work is concerned with the
              management of services
Open Grid Services Architecture                                    6
                         OGSA Structure
         A standard substrate: the Grid service
             Standard interfaces and behaviors that
              address key distributed system issues:
              naming, service state, lifetime, notification
             A Grid service is a Web service
         … supports standard service specifications
             Agreement, data access & integration,
              workflow, security, policy, diagnostics, etc.
             Target of current & planned GGF efforts
         … and arbitrary application-specific services
          based on these & other definitions
Open Grid Services Architecture                               7
                                  Overview
         Grid background
         Open Grid Services Architecture
         Open Grid Services Infrastructure
         Beyond OGSI: other OGSA services
         Globus Toolkit v3 implementation
         Early GT3 performance results
         Scientific and commercial perspectives
         Summary


Open Grid Services Architecture                    8
                      OGSI Specification
         Defines WSDL conventions and extensions
             For describing and naming services
             Working with W3C WSDL working group to
              drive OGSI extensions into WSDL 1.2
         Defines fundamental interfaces (using
          extended WSDL) and behaviors that define a
          Grid Service
             A unifying framework for interoperability &
              establishment of total system properties
         http://www.ggf.org/ogsi-wg
Open Grid Services Architecture                             9
                         Fundamental
                    Interfaces & Behaviors
         OGSI defines basic patterns of interaction,
          which can be combined with each other and
          with custom patterns in a myriad of ways
         OGSI Specification focuses on:
             Atomic, composable patterns in the form of
              portTypes/interfaces
                  Define operations & associated service data elements
             A model for how these are composed
                  Compatible with WSDL 1.2

         Complete service descriptions are left to
          other groups that are defining real services
Open Grid Services Architecture                                           10
           OGSI: Standard Web Services
             Interfaces & Behaviors
         Naming and bindings (basis for virtualization)
             Every service instance has a unique name, from which
              can discover supported bindings
         Lifecycle (basis for fault resilient state management)
             Service instances created by factories
             Destroyed explicitly or via soft state
         Information model (basis for monitoring & discovery)
             Service data (attributes) associated with GS instances
             Operations for querying and setting this info
             Asynchronous notification of changes to service date
         Service Groups (basis for registries & collective svcs)
             Group membership rules & membership management
      Base Fault type
Open Grid Services Architecture                                        11
                      Open Grid Services
                     Infrastructure (OGSI)
Resource
allocation
                                        Create
                                        Service
Authentication
                                 Grid Service
& authorization      Service                        Service
                                 Handle
are applied to       factory                        requestor
all requests                                        (e.g. user
                                Service data        application)
                                Keep-alives
                                Notifications
                                Service           Service
                                invocation        discovery


                                       Register
      Service                          Service         Service
      instances                                        registry


Interactions standardized using WSDL
Open Grid Services Architecture                               12
       Open Grid Services Infrastructure
                         Client
        Introspection:
                                  Lifetime management
        • What port types?
                                  • Explicit destruction
        • What policy?
                                  • Soft-state lifetime
        • What state?
                        GridService       Data
                        (required)       access    Other standard interfaces:
Grid Service                                                          factory,
  Handle                                                         notification,
                          Service        Service     Service      collections
       handle               data           data        data
                          element        element     element
       resolution

Grid Service
 Reference
                                  Implementation


                         Hosting environment/runtime
                              (“C”, J2EE, .NET, …)
Open Grid Services Architecture                                                  13
                                  GWSDL
         OGSI requires interface extension/composition
         We worked within W3C WSDL working group to define
          standard interface extension in WSDL 1.2 that meets
          OGSI requirements
         But could not wait for WSDL 1.2
         So defined gwsdl:portType that extends WSDL 1.1
          portType with:
             WSDL 1.2 portType extension
             WSDL 1.2 open content model
         Define GWSDL  WSDL 1.1 & 1.2 mappings



Open Grid Services Architecture                             14
                        GWSDL Example
       <wsdl:definitions>
        <wsdl:types>…</wsdl:types>
        <wsdl:message>…</wsdl:message>
        …
        <gwsdl:portType name=“foo”
                          extends=“ns:bar ogsi:GridService”>

            <wsdl:operation name=“op1”>…</wsdl:operation>
            <wsdl:operation name=“op2”>…</wsdl:operation>

            <ogsi:serviceData … />

        </gwsdl:portType>
        …
       </wsdl:definitions>

Open Grid Services Architecture                             15
                               OGSI portTypes
    portType name               Description


    <TB>GridService             Encapsulates the root behavior of the service model,
                                including service data and lifetime management
    HandleResolver              Mapping from a GSH to a GSR
    NotificationSource          Allows clients to subscribe to notification messages
    NotificationSubscription    For managing the lifetime and other properties of a
                                subscription
    NotificationSink            Defines a single operation for delivering a notification
                                message to the service instance that implements the
                                operation
    Factory                     Standard operation for creation of Grid service instances
    ServiceGroup                Service data for representing a group of service instances,
                                and the policies associated to group membership
    ServiceGroupRegistration    Allows Grid services to be added and removed from a
                                ServiceGroup
    ServiceGroupEntry           For managing the lifetime and other properties of a
                                registration to a ServiceGroup</TB>

Open Grid Services Architecture                                                               16
        GridService portType Operations


 Operation                        Description

 <TB>findServiceData              Query information about the Grid
                                  service instance
 setServiceData                   Modify service data values
 requestTerminationAfter          Specify earliest desired
                                  termination time
 requestTerminationBefore         Specify latest desired termination
                                  time
 destroy                          Terminate Grid service instance




Open Grid Services Architecture                                        17
                        OGSI Service Data
         Attributes: Publicly visible state of the service
         Want to bring full power of XML to attributes
             getXXX/setXXX is too limiting
                  How to get/set multiple?
                  Want richer queries across attributes (e.g., join)
             Use XML Schema, XPath, XQuery, XSLT, etc.
             OGSI service data:
                  Attributes defined using XML Schema
                  Attributes combined into a single (logical) document within
                   the service
                  Rich pull/push/set operations against service data document

         Should declare attributes in WSDL interface
Open Grid Services Architecture                                                  18
        Standard Service Data Elements
 SDE                                   Description

 Interface                        1+   Names of the interfaces (portTypes)
                                       supported by the Grid service instance
 ServiceDataName                  0+   Names of the service data elements
                                       contained in the instance’s service data
                                       values document
 FactoryLocator                   1    The factory that created the instance, or
                                       NIL if none
 GridServiceHandle                0+   GSHs for the Grid service instance
 GridServiceReference             1+   GSRs for the Grid service instance
 FindServiceDataExtensibility     1+   Query expression types that can be
                                       passed to the findServiceData operation
 SetServiceDataExtensibility      2+   Update expression types that can be
                                       passed to the setServiceData operation
 TerminationTime                  1    Earliest and latest termination times of
                                       the Grid service instance

Open Grid Services Architecture                                                    19
                  Handles and Resolvers

                                  resolve(GSH)      Handle
                     Client
                                                   Resolver
                                  time > T
                   time < T
                    GSR1          GSR2


                                        Migrate
                        Service                       Service
                       Instance        at time T     Instance




Open Grid Services Architecture                                 20
                          Service Groups




                 portType GridService
                 portType Factory               Service
                 portType ServiceGroup         Instance
                 SDE Entry        SDE Entry
                   GSH              GSH
                                                Service
                  Other info      Other info   Instance


Open Grid Services Architecture                           21
                      Example:
            Reliable File Transfer Service

                                  Client          Client             Client


                             Request and manage file transfer operations
                                   Grid     File   Notf’n Policy
   Fault                          Service Transfer Source
  Monitor                           Pending
                Query &/or                              interfaces
                 subscribe        Performance
                                                  service
              to service data       Policy        data
                                                             Internal
   Perf.                                          elements    State
  Monitor                           Faults


                                           Data transfer operations
Open Grid Services Architecture                                               22
                    File Transfer Example


                                                  Storage Provider
                              Storage Monitor         Service
                                  Service

                                                         .
                                  File Transfer          .
                 Client
                                     Service             .
                                                  Storage Provider
                                                      Service
                              Storage Broker
                                  Service




Open Grid Services Architecture                                      23
                               File Transfer Factories
                                                                    ReliableTransferService

                                                           Interfaces:
                                                               Managemernt, TransferInstanceFactory

                                                           State:
                                                               <ActiveTransfers 0>                                 Storage Service

                                                                                                      Interfaces:
                                                                                                          SLAFactory, TransferFactory

                                                                                                      State:
                                                                                                          <Name ReliableStorage>
                                                                                                          <Storage Type RAID5,UPS>
                            Storage VO                                StorageBrokerService                <Services supported mgmt, transfer>
                                                                                                          <PeekTransferRate 100MB/sec>
          Interfaces:                                       Interfaces:                                   <MaxStorageSpace 10TB>
              Managemernt, Index                                TransferPathFactory                       <ResearvableSpace 3TB>
                                                                                                          <ReservableBandwidth 50MB/sec>
          State:                                            State:
              <ServiceCollection < >                            <Status Up>
              <InfoCache                                        <ManagedStorage < > >                              Storage Service
                  <StorageService
                      <Name ReliableStorage>                                                          Interfaces:
                      <MaxStorageSpace 10TB>>                                                             SLAFactory, TransferFactory
                  <StorageService
                      <Name DataCache>                                                                State:
                      <MaxStorageSpace 100TB>>                                                            <Name DataCache>
                  <MonitorTransferService                                                                 <Storage Type parallel transfer>
                      <MonitorServices FileTransfer,SLA>            MonitorTransferService                <Services supported mgmt, transfer>
              <ReliableTransferService <Status Up>                                                        <PeekTransferRate 500MB/sec>
                                                           Interfaces:                                    <MaxStorageSpace 100TB>
                                                               Managemernt, MonitorFactory                <ResearvableSpace 35TB>
                                                                                                          <ReservableBandwidth 200MB/sec>
                                                           State:
                                                               <MonitorType notify,poll>
                                                               <MonitorServices FileTransfer,SLA>




Open Grid Services Architecture                                                                                                                 24
                           Making a Reservation
                                                                                                    Storage Service

                                                                                      Interfaces:
                                                                                          SLAFactory, TransferFactory

                                                                                      State:
                                                                                           <Name ReliableStorage>
                                                                                           <Storage Type RAID5,UPS>
                                                                           StorageReservation
                                                                                           <Services supported mgmt, transfer>
                                        StorageBrokerService   Interfaces:                 <PeekTransferRate 100MB/sec>
                                                                   SLAManagemernt          <MaxStorageSpace 10TB>
                              Interfaces:                                                  <ResearvableSpace 3TB>
                                  TransferInstanceFactory      State:                      <ReservableBandwidth 50MB/sec>
                        StorageReservation                         <SLA-1 RSLA ReliableStorage
                            State:                                         <DiskBandwidth 2MB/sec>>
          Interfaces:
                                <Status Up>
              SLAManagement, Index
                                <ManagedStorage         >
                                                                                                    Storage Service
          State:
              <ServiceCollection < >                                                  Interfaces:
              <InfoCollection                                                             SLAFactory, TransferFactory
                  <SLA-1 RSLA ReliableStorage
                     <DiskBandwidth 2MB/sec>                                          State:
                  <SLA-2 RSLA DataCache                                                    <Name DataCache>
                                                                           StorageReservation
                     <DiskSpace 3TB>>                                                      <Storage Type parallel transfer>
                                                               Interfaces:                 <Services supported mgmt, transfer>
                                                                   SLAManagement           <PeekTransferRate 100MB/sec>
                                                                                           <MaxStorageSpace 100TB>
                                                               State:                      <ResearvableSpace 35TB>
                                                                   <SLA-2 RSLA DataCache   <ReservableBandwidth 200MB/sec>
                                                                           <DiskSpace 3TB>>




Open Grid Services Architecture                                                                                                  25
               Establishing Delivery Service
                                  ReliableTransferService
                                                                                                       Storage Service
                        Interfaces:
                  DataTransfer                                                            Interfaces:
                            Managemernt, TransferInstanceFactory
                                                                                              Managemernt, TransferFactory

        Interfaces:      State:
                                                                                         State:
            DataTransfer     <ActiveTransfers 1>
                                                                                             <Name ReliableStorage>
                                                                               GridFTPService<Storage Type RAID5,UPS>
        State:                                                                               <Services supported mgmt, transfer>
            <Bandwidth 1.5MB/sec>                                  Interfaces:               <PeekTransferRate 100MB/sec>
            <PercentComplete 75>                                       SLAManagemernt        <MaxStorageSpace 10TB>
            <Source< >>                                                                      <ResearvableSpace 3TB>
            <Dest < >>>                                            State:                    <ReservableBandwidth 50MB/sec>
                                                                       <PercentComlete 75>
                                                                       <Bandwidth 2MB/sec>


                                                                                                       Storage Service

                                                                                          Interfaces:
                                                                                              Managemernt, TransferFactory

                                                                                         State:
                                                                               GridFTPService<Name DataCache>
                                                                                             <Storage Type parallel transfer>
                                                                   Interfaces:               <Services supported mgmt, transfer>
                                                                       SLAManagement         <PeekTransferRate 500MB/sec>
                                                                                             <MaxStorageSpace 100TB>
                                                                   State:                    <ResearvableSpace 35TB>
                                                                       <PercentComplete 74> <ReservableBandwidth 200MB/sec>
                                                                       <Bandwidth 2MB/sec-2>




Open Grid Services Architecture                                                                                                    26
                                    Monitoring Progress

                              MonitorTransferService                        StorageReservation

                                                              Interfaces:
                    Interfaces:                                   Managemernt, Index
                        Management, MonitorFactory
                                                              State:
               TaskMonitor
                     Status:                                      <ServiceCollection
                         <MonitorType notify,poll>                <InfoCollection
     Interfaces:         <MonitorServices FileTransfer,SLA>           <SLA-1 RSLA ReliableStorage
         Monitor, Index                                                  <DiskBandwidth 2MB/sec>
                                                                      <SLA-2 RSLA DataCache
     State:                                                              <DiskSpace 3TB>>
         <ServiceCollection < > >
         <Status OK>

                                                                        DataTransfer


                                                              Interfaces:
                                                                  DataTransfer

                                                              State:
                                                                  <Bandwidth 1.5MB/sec>
                                                                  <PercentComplete 75>
                                                                  <Source >
                                                                  <Dest   >>




Open Grid Services Architecture                                                                     27
                  OGSI Implementations
         Globus Toolkit version 3.0 (Java, C client)
         U Virginia OGSI.NET (.NET)
         LBNL pyGlobus (Python)
         U Edinburgh (.NET)
         U Manchester (PERL)
         Fujitsu Unicore (Java)




Open Grid Services Architecture                         28
                                  Overview
         Grid background
         Open Grid Services Architecture
         Open Grid Services Infrastructure
         Beyond OGSI: other OGSA services
         Globus Toolkit v3 implementation
         Early GT3 performance results
         Scientific and commercial perspectives
         Summary


Open Grid Services Architecture                    29
        Open Grid Services Architecture
                        Users in Problem Domain X

                     Applications in Problem Domain X
       Application & Integration Technology for Problem Domain X
          Generic Virtual Service Access and Integration Layer
    Job Submission      Brokering        Workflow        Structured Data
       Registry          Banking
                                OGSA
                                  Authorisation            Integration

    Data Transport Resource Usage Transformation Structured Data Access

                  OGSI: Interface to Grid Infrastructure
                        Web Services: Basic Functionality

         Compute, Data & Storage Resources
                                                        Structured Data
                                                    Relational XML      -
                                                                     Semi-structured
                                   Distributed
                     Virtual
Open Grid Services Architecture Integration Architecture                         30
                   OGSA Standardization
                    & Implementation
        OGSI defines core interfaces and behaviors
         for manageable services
            Supported by strong open source technology
             & major commercial vendors
        Efforts are underway within GGF, OASIS, and
         other bodies to define standards for
            Agreement negotiation
            Common management model
            Data access and integration
            Security and policy
            Etc., etc., etc.
Open Grid Services Architecture                           31
                 Transactions & Contexts
         WS-Coordination & WS-Transaction
             IBM/MS (not in standards org)
         WS-CAF (Coordinated Application Framework)
             Sun/Oracle/Arjuna/Fujitsu (not in standards org)
             WS-CTX (Context)
             WS-CF (Coordination Framework)
             WS-TXM (Transaction Management)
         Both take a “contextualization” approach
             Context (id) threaded through SOAP header
             OGSI for context creation, naming & lifecycle???
Open Grid Services Architecture                             32
                       Security Standards
         Many core security standards are from IETF
             X.509, Kerberos, etc.
             X.509 Proxy Certificates (RFC soon hopefully)
                  Used by Globus Toolkit GSI

         OASIS appears to be leader in Web services
          security standards
             WS-Security: SOAP message security
             SAML: signed assertions using XML
             XACML: access control lists using XML
         GGF OGSA Security WG evaluating security
          specifications for applicability to OGSA
Open Grid Services Architecture                               33
                    IBM/Microsoft
                WS Security Architecture
         Large set of specifications for doing Web
          services security, most of which should be
          appropriate for OGSA
         Announced April 2002
         Initial spec in July 2002 (WS-Security)
             Submitted to OASIS
         New crops of specs arrive periodically
             WS-Policy*, WS-Trust, WS-Federation, etc.
             But… Not yet in any standards organization


Open Grid Services Architecture                            34
                   WS Security
           Current/Proposed WSS-specs


             WS-Secure
                              WS-Federation   WS-Authorization
            Conversation



              WS-Policy           WS-Trust      WS-Privacy



                             WS-Security

                                                                 In progress

                           SOAP Foundation                       proposed

                                                                 promised

Open Grid Services Architecture                                                35
                  OASIS SAML & XACML
         SAML: Security Assertion Markup Language
             Good for asserting properties such as group
              membership, etc
         XACML: eXtensible Access Control Markup
          Language
             For defining access control policies
         These are gaining considerable momentum,
          but WS-Policy* leaves them in question



Open Grid Services Architecture                             36
                  Project Liberty Alliance
         V1.x specifications for identity federation
             Allows cross-organization identification
             Privacy preserving model
         Based on SAML




Open Grid Services Architecture                          37
                          WS-Agreement
         Recall key criteria of a Grid:
             Coordinates resources that are not subject to
              centralized control …
             using standard, open, general-purpose
              protocols and interfaces …
             to deliver non-trivial qualities of service.
         Implies need to express and negotiate
          agreements that govern the delivery of
          services to clients
             Agreement = what will be done, QoS, billing,
              compliance monitoring
Open Grid Services Architecture                               38
                WS-Agreement Contents
         Standard agreement language
             A composition of a set of terms that govern a
              service’s behavior with respect to clients
             Agreement language uses WS-Policy (currently)
             Standard attributes for terms that express
              current state of negotiation
             Other groups define specific terms
         Standard agreement negotiation protocol
             Establish, monitor, re-negotiate agreement
             Expressed using OGSI GWSDL interfaces
             Each agreement represented by a service
Open Grid Services Architecture                               39
              WS-Agreement Applicability
         All interesting Web/Grid services interactions
          will be governed by agreements!
         WS-Agreement (language and interfaces)
          should be used by specifications that define
          domain-specific services
             Data services
             Job submission
             Specialized services
             Etc.


Open Grid Services Architecture                            40
       Platform’s Community Scheduling
         Framework & WS-Agreement

            GT3.0                              GT3.0
          Queuing                                                 LSF
                                             RM Adapter
           Service
                                  Internet
            Job
           Service                                       Site B
         Reservation
           Service
                                               GT3.0
                                                                  PBS
            Site A                           RM Adapter



                                                         Site C

        Reservation Agreement Exchange         GT3.0
                                                 RM               SGE
                                               Adaptor



Open Grid Services Architecture                                         41
                     WSDM / WSMF / CMM
         OASIS Web Services Distributed Management
          (WSDM) technical committee
             Management using/of Web Services
             HP submitted its Web Services Management
              Framework (WSMF) to WSDM in July 2003
                  WS-Events: event schema, subscription, message queues
                  WSMF-Foundation: management using Web services
                  WSM: management of Web services

         GGF Common Management Model (CMM) WG
             IBM submission overlaps WSMF-Foundation
         Working to bring WSDM & CMM together,
          based on OGSI foundation
Open Grid Services Architecture                                            42
                Data as Service:
          OGSA Data Access & Integration
         Service-oriented treatment of data appears
          to have significant advantages
             Leverage OGSI introspection, lifetime, etc.
             Compatibility with Web services
         Standard service interfaces being defined
             Service data: e.g., schema
             Derive new data services from old (views)
             Externalize to e.g. file/database format
             Perform queries or other operations


Open Grid Services Architecture                             43
                           Data Services
         GGF Data Access and Integration Svcs (DAIS)
             OGSI-compliant interfaces to access relational
              and XML databases
             Needs to be generalized to encompass other
              data sources (see next slide…)
         Generalized DAIS becomes the foundation for:
             Replication: Data located in multiple locations
             Federation: Composition of multiple sources
             Provenance: How was data generated?


Open Grid Services Architecture                                 44
              “OGSA Data Services”
           (Foster, Tuecke, Unger, eds.)
         Describes conceptual model for representing
          all manner of data sources as Web services
             Database, filesystems, devices, programs, …
             Integrates WS-Agreement
         Data service is an OGSI-compliant Web
          service that implements one or more of base
          data interfaces:
             DataDescription, DataAccess, DataFactory,
              DataManagement
             These would be extended and combined for
              specific domains (including DAIS)
Open Grid Services Architecture                             45
        Data Access & Integration Services
               1a. Request to Registry
               for sources of data
               about “x”
                                              Registry                              SOAP/HTTP
                                                                                    service creation
                               1b. Registry
                            responds with                                           API interactions
                           Factory handle
                                              2a. Request to Factory for access
                                              to database


                                                                   Factory
                                2c. Factory returns
          Client                handle of GDS to
                                client                     2b. Factory creates
                     3a. Client queries GDS with           GridDataService to manage
                                  XPath, SQL, etc          access

                                                                                  XML /
                                                                                  Relationa
                                                    Grid Data                     l
                                                    Service                       database
        3c. Results of query returned to
        client as XML
                                                      3b. GDS interacts with database
            Slide Architecture
Open Grid Services Courtesy Malcolm Atkinson, UK eScience Center                                  46
                                  Overview
         Grid background
         Open Grid Services Architecture
         Open Grid Services Infrastructure
         Beyond OGSI: other OGSA services
         Globus Toolkit v3 implementation
         Early GT3 performance results
         Scientific and commercial perspectives
         Summary


Open Grid Services Architecture                    47
                     The Globus Alliance
          A group of people with a common mission:
              “Make Grid computing an everyday reality.”
          Argonne/U.Chicago, USC-ISI, EPCC, & KTH-PDC
              Led by Ian Foster (Argonne), Carl Kesselman (ISI),
               Malcolm Atkinson (EPCC), Lennart Johnsson (PDC)
              Includes researchers, software developers, software
               architects & designers, systems engineers, & others
              Collaborations (or at least acquaintances) with most
               Grid activities in the world
          All activities contribute to our common mission
              Research
              Software development (prototypes, reference implns)
              Application consulting
              Infrastructure consulting
Open Grid Services Architecture                                       48
                       Globus Toolkit:
                     A Story of Evolution
       Definition of Grid problem has been stable
        since original Globus Project proposal in 1995
           Though we’ve gotten better at articulating it
       But our approach to its solution has evolved:
           From APIs and custom protocols…
           to standard protocols…
           to Grid services (OGSA)
       Driven by experience implementing and
        deploying the Globus Toolkit, and building
        real applications with it
Open Grid Services Architecture                             49
                    Globus Toolkit® v3.0
         All of the GT v2.4 services and clients
         Complete Java implementation of OGSI v1.0
             Rich, container-based implementation
             Built on Apache Axis
         Globus “proprietary” services built on OGSI
             Managed Jobs (akin to GT2 GRAM)
             Reliable File Transfer (RFT)
             Index Services (akin to GT2 GIIS)
         Some services not yet OGSI-fied:
             GridFTP, Replica Location Services (RLS)
Open Grid Services Architecture                          50
                         GT3 Distribution




                                                GT-OGSA
              GT2
                                  RLS          Grid Service
           Components
                                              Infrastructure




                                    The focus of this presentation

Open Grid Services Architecture                                      51
        GT-OGSA Grid Service Infrastructure

                          Grid Service Container

                            User-Defined Services

                                  Base Services

                          System-Level Services


     OGSI Spec Implementation              Security Infrastructure


                            Web Service Engine

                           Hosting Environment

Open Grid Services Architecture                                      52
        GT-OGSA Grid Service Infrastructure

                          Grid Service Container

                            User-Defined Services

                                  Base Services

                          System-Level Services


     OGSI Spec Implementation              Security Infrastructure


                            Web Service Engine

                           Hosting Environment

Open Grid Services Architecture                                      53
                   OGSI Implementation
         GT3 includes a set of primitives that fully
          implement the interfaces and behaviors
          defined in the OGSI Specification
             Defines how entities can create, discover
              and interact with a Grid service
         The OGSI Specification defines a protocol:
          GT3 provides a programming model for
          that protocol




Open Grid Services Architecture                           54
             Implementation of the GridService
               portType: GridServiceImpl and
                 PersistentGridServiceImpl
      destroy()                               Implementation
                                               of the OGSI Spec
      setServiceData()
      findServiceData()
      requestTerminationAfter()
                                              Additional
      requestTerminationBefore()             functionality can be
                                              added to a
      addOperationProvider()                 Grid Service using
   (See docs for complete set of methods)     OperationProviders


 Deployment descriptor
 <parameter name=“operationProviders” value=“<className>”>

Open Grid Services Architecture                                      55
             Building an OGSI-Compliant
               Grid Service using GT3




 Write service-specific
 logic that also
 implements the GT3
 OperationProvider
 interface



Open Grid Services Architecture           56
             Building an OGSI-Compliant
               Grid Service using GT3




 Write service-specific     Combine with one of the
 logic that also            two GT3 implementations
 implements the GT3         of base GridService
 OperationProvider          functionality:
 interface                  GridServiceImpl or
                            PersistentGridServiceImpl


Open Grid Services Architecture                         57
             Building an OGSI-Compliant
               Grid Service using GT3




 Write service-specific     Combine with one of the            An
 logic that also            two GT3 implementations     OGSI-Compliant
 implements the GT3         of base GridService           grid service
 OperationProvider          functionality:
 interface                  GridServiceImpl or
                            PersistentGridServiceImpl


Open Grid Services Architecture                                          58
             Building an OGSI-Compliant
               Grid Service using GT3




                                  OperationProviders are
                                  configured at deployment time
                                  or added at runtime



 Write service-specific     Combine with one of the              An
 logic that also            two GT3 implementations       OGSI-Compliant
 implements the GT3         of base GridService             grid service
 OperationProvider          functionality:
 interface                  GridServiceImpl or
                            PersistentGridServiceImpl


Open Grid Services Architecture                                            59
        A Grid Service Can be Composed
         of Multiple OperationProviders
    OPs can be designed as
     atomic bits of functionality
     to facilitate reuse
    OP approach eases the
     task of bringing legacy
     code into OGSI-compliance
    OPs allow Grid Services to
     be formed dynamically (in
     contrast to the inheritance
     approach)
Open Grid Services Architecture           60
            Several OperationProviders
       are Included in the GT3 Distribution
 NotificationSourceProvider

                  HandleResolverProvider

                                  ServiceGroupProvider

                                           ServiceGroupRegistrationProvider

                                                         FactoryProvider




Open Grid Services Architecture                                            61
          OGSI Specification: Notifications
         Our NotificationSourceProvider
          implementation allows any Grid Service to
          become a sender of notification messages
         A subscribe request on a
          NotificationSource triggers the creation of
          a NotificationSubscription service
         A NotificationSink can receive notification
          msgs from NotificationSources
             Sinks are not required to implement the
              GridService portType
         Notifications can be set on SDEs
Open Grid Services Architecture                         62
               OGSI Specification: Factory

       Factory portType
          Factories create services
          Factories are typically persistent services
          Factory is an optional OGSI interface


       (Grid Services can also be instantiated by
         other mechanisms)


Open Grid Services Architecture                          63
       OGSI Specification: Service Groups
  Service group portTypes
     A ServiceGroup is a grid service that maintains
      information about a group of other grid services
     The classic registry model can be implemented with the
      ServiceGroup portTypes
     A grid service can belong to more than one ServiceGroup
     Members of a ServiceGroup can be heterogeneous or
      homogenous
     Each entry in a service group can be represented as its
      own service
     Service group portTypes are optional OGSI interfaces

Open Grid Services Architecture                                 64
       OGSI Specification: Handle Resolver
   HandleResolver portType
      Defines a means for resolving a GSH (Grid Service
       Handle) to a GSR (Grid Service Reference)
           A GSH points to a Grid Service
             (GT3 uses a hostname-based GSH scheme)
           A GSR specifies how to communicate with the Grid Service
             (GT3 currently supports SOAP over HTTP, so GSRs are in WSDL format)




   Service locator
     Includes 0 or more Grid Service Handles (GSHs)
     Includes 0 or more Grid Service References (GSRs)

Open Grid Services Architecture                                                65
             A Service Creation Scenario*

               Registry


 1. From a
 known registry,
 the client
 discovers a
 factory by
 querying the
 service data
 of the
 registry
                Client



* All scenarios are offered as examples and are not prescriptive
Open Grid Services Architecture                                    66
             A Service Creation Scenario

               Registry
                                                        Factory
                                  2. The client calls
                                  the createService
 1. From a                        operation on the
 known registry,                  factory
 the client
 discovers a
 factory by
 querying the
 service data
 of the
 registry
                Client




Open Grid Services Architecture                                   67
             A Service Creation Scenario

               Registry
                                                        Factory
                                  2. The client calls
                                  the createService
 1. From a                        operation on the
 known registry,                  factory                   3. The
 the client                                                 factory
 discovers a                                                creates a
 factory by                                                 service
 querying the
 service data
 of the
 registry
                Client                                  Service




Open Grid Services Architecture                                         68
             A Service Creation Scenario

               Registry
                                                            Factory
                                  2. The client calls
                                  the createService
 1. From a                        operation on the
 known registry,                  factory                       3. The
 the client                                                     factory
 discovers a                         4. The factory             creates a
 factory by                          returns a locator to       service
 querying the                        the newly created
 service data                        service
 of the
 registry
                Client                                      Service




Open Grid Services Architecture                                             69
             A Service Creation Scenario

               Registry
                                                            Factory
                                   2. The client calls
                                   the createService
 1. From a                         operation on the
 known registry,                   factory                      3. The
 the client                                                     factory
 discovers a                         4. The factory             creates a
 factory by                          returns a locator to       service
 querying the                        the newly created
 service data                        service
 of the
 registry
                Client                                      Service
                                  5. The client and
                                  service interact



Open Grid Services Architecture                                             70
                    A Notification Scenario

                                                    Notification
                                                    Source
                        1. NotificationSink calls
                        the subscribe operation
                        on NotificationSource




           Notification
           Sink



Open Grid Services Architecture                                    71
                    A Notification Scenario

                                                    Notification
                                                    Source
                        1. NotificationSink calls
                        the subscribe operation
                        on NotificationSource
                                                         2.Notification
                                                         Source
                                                         creates a
                                                         subscription
                                                          service



           Notification                             Notification
           Sink                                     Subscription



Open Grid Services Architecture                                       72
                    A Notification Scenario

                                                            Notification
                                                            Source
                        1. NotificationSink calls
                        the subscribe operation
                        on NotificationSource
                                                                 2.Notification
                                                                 Source
                                          3. Notification        creates a
                                     Source returns a            subscription
                                 locator to the                   service
                             subscription service


           Notification                                     Notification
           Sink                                             Subscription



Open Grid Services Architecture                                               73
                    A Notification Scenario
          4.a deliverNotification
          stream continues
          for the lifetime of
          NotificationSubscription                          Notification
                                                            Source
                        1. NotificationSink calls
                        the subscribe operation
                        on NotificationSource
                                                                 2.Notification
                                                                 Source
                                          3. Notification        creates a
                                     Source returns a            subscription
                                 locator to the                   service
                             subscription service


           Notification                                     Notification
           Sink            4.b The NotificationSink and     Subscription
                           Subscription service interact
                           to perform lifetime
                           management

Open Grid Services Architecture                                               74
                    A Notification Scenario
          4.a deliverNotification
          stream continues            The sole mandated
          for the lifetime of         cardinality: 1 to 1
          NotificationSubscription                          Notification
                                                            Source
                        1. NotificationSink calls
                            subscribe
                        the subscribe operation
                        on NotificationSource
                                                                 2.Notification
                                                                 Source
                                          3. Notification        creates a
                                     Source returns a            subscription
                                 locator to the                   service
                             subscription service


            Notification                                    Notification
            Sink           4.b The NotificationSink and     Subscription
                           Subscription service interact
                           to perform lifetime
                           management

Open Grid Services Architecture                                               75
        GT-OGSA Grid Service Infrastructure

                          Grid Service Container

                            User-Defined Services

                                  Base Services

                          System-Level Services


     OGSI Spec Implementation             Security Infrastructure


                            Web Service Engine

                           Hosting Environment

Open Grid Services Architecture                                     76
                    Security Infrastructure
          Transport Layer Security/Secure Socket
           Layer (TLS/SSL)
              To be deprecated
          SOAP Layer Security
              Based on WS-Security, XML Encryption, XML
               Signature
          GT3 uses X.509 identity certificates for
           authentication
          It also uses X.509 Proxy certificates to
           support delegation and single sign-on,
           updated to conform to latest IETF/GGF draft

Open Grid Services Architecture                            77
        GT-OGSA Grid Service Infrastructure

                          Grid Service Container

                            User-Defined Services

                                  Base Services

                         System-Level Services


     OGSI Spec Implementation              Security Infrastructure


                            Web Service Engine

                           Hosting Environment

Open Grid Services Architecture                                      78
                   System-Level Services

      General-purpose services that facilitate the use
       of Grid services in production environments
      The 3.0 distribution includes three system-level
       services
           An administration service
           A logging service
           A management service




Open Grid Services Architecture                       79
        GT-OGSA Grid Service Infrastructure

                          Grid Service Container

                            User-Defined Services

                                  Base Services

                          System-Level Services


     OGSI Spec Implementation              Security Infrastructure


                            Web Service Engine

                           Hosting Environment

Open Grid Services Architecture                                      80
                  Grid Service Container
      Includes the OGSI Implementation, security
      infrastructure and system-level services, plus:
     Service activation, deactivation, construction,
      destruction, etc.
     Service data element placeholders that allow
      you to fetch service data values dynamically, at
      query time
     Evaluator framework (supporting ByXPath and
      ByName notifications and queries)
     Interceptor/callback framework (allows one to
      intercept certain service lifecycle events)
Open Grid Services Architecture                          81
           Grid Service Container (cont.)

                  Layers in the Web Services Model


                   Interface Layer           OGSI Spec is here

                                             Transport/binding
                  Transport Layer            layer (e.g., GT3:
                                             SOAP over HTTP)


              Implementation Layer           Container is here



Open Grid Services Architecture                              82
        GT-OGSA Grid Service Infrastructure

                          Grid Service Container

                            User-Defined Services

                                  Base Services

                          System-Level Services


     OGSI Spec Implementation              Security Infrastructure


                            Web Service Engine

                         Hosting Environment

Open Grid Services Architecture                                      83
                     Hosting Environment

             GT3 currently offers support for four
             Java hosting environments:
            Embedded
            Standalone
            Servlet
            EJB




Open Grid Services Architecture                      84
             Virtual Hosting Environment
                      Framework
        Virtual hosting allows Grid services to be
         distributed across several remote containers
        Useful in implementing solutions for
         problems common to distributed computing
            Load balancing
            User account sandboxing




Open Grid Services Architecture                         85
        GT-OGSA Grid Service Infrastructure

                          Grid Service Container

                            User-Defined Services

                                  Base Services

                          System-Level Services


     OGSI Spec Implementation             Security Infrastructure


                            Web Service Engine

                           Hosting Environment

Open Grid Services Architecture                                     86
                   Resource Management
         GRAM Architecture rendered in OGSA
         The MMJFS runs as an unprivileged user,
          with a small highly-constrained setuid
          executable behind it
         Individual user environments are created
          using virtual hosting
                                                   MMJFS: Master
                                             MJS   Managed Job Factory
                                  User 1    MJS    Service
              Master User                  MJS
                                                   MJS: Managed Job
              MMJFS               User 2   MJS     Service

                                  User 3
                                           MJS         User Hosting
                                            MJS        Environment
Open Grid Services Architecture                                       87
          GRAM Job Submission Scenario
                Index
                Service

   1. From an
   index service,
   the client
   chooses
   an MMJFS




                Client




Open Grid Services Architecture          88
          GRAM Job Submission Scenario
                Index
                Service      2. The client calls the
                                                        MMJFS
                             createService
                             operation on the factory
   1. From an                and supplies RSL
   index service,
   the client
   chooses
   an MMJFS




                Client




Open Grid Services Architecture                                 89
          GRAM Job Submission Scenario
                Index
                Service      2. The client calls the
                                                        MMJFS
                             createService
                             operation on the factory
   1. From an                and supplies RSL
                                                          3. The factory
   index service,
                                                          creates a
   the client
                                                          Managed Job
   chooses
                                                          Service
   an MMJFS




                Client                                  MJS




Open Grid Services Architecture                                        90
          GRAM Job Submission Scenario
                Index
                Service      2. The client calls the
                                                         MMJFS
                             createService
                             operation on the factory
   1. From an                and supplies RSL
                                                           3. The factory
   index service,
                                                           creates a
   the client
                                                           Managed Job
   chooses
                                                           Service
   an MMJFS                           4. The factory
                                     returns a locator


                Client                                   MJS




Open Grid Services Architecture                                         91
          GRAM Job Submission Scenario
                Index
                Service      2. The client calls the
                                                          MMJFS
                             createService
                             operation on the factory
   1. From an                and supplies RSL
                                                            3. The factory
   index service,
                                                            creates a
   the client
                                                            Managed Job
   chooses
                                                            Service
   an MMJFS                           4. The factory
                                     returns a locator


                Client                                    MJS
                            5. The client subscribes to
                            the MJS’ status SDE and
                            retrieves output


Open Grid Services Architecture                                          92
                    Information Services
         Index service as caching aggregator
             Caches service data from other Grid
              services
         Index service as provider framework
             Serves as a host for service data providers
              that live outside of a Grid service to publish
              data




Open Grid Services Architecture                                93
                    Reliable File Transfer
         OGSI-compliant service exposing GridFTP
          control channel functionality
             3rd-party transfer between GridFTP servers
         Recoverable Grid service
             Automatically restarts interrupted transfers
              from the last checkpoint
         Progress and restart monitoring
                                                   GridFTP
                                                   Server 1
                                  RFT

                                                   GridFTP
                                  JDBC             Server 2
Open Grid Services Architecture                               94
            Reliable File Transfer Service

                                  Client          Client             Client


                             Request and manage file transfer operations
                                   Grid     File   Notf’n Policy
   Fault                          Service Transfer Source
  Monitor                           Pending
                Query &/or                              interfaces
                 subscribe        Performance
                                                  service
              to service data       Policy        data
                                                             Internal
   Perf.                                          elements    State
  Monitor                           Faults


                                           Data transfer operations
Open Grid Services Architecture                                               95
        GT-OGSA Grid Service Infrastructure

                          Grid Service Container

                            User-Defined Services

                                  Base Services

                          System-Level Services


     OGSI Spec Implementation              Security Infrastructure


                            Web Service Engine

                           Hosting Environment

Open Grid Services Architecture                                      96
                GT3 User-Defined Services
         GT3 can be viewed as a Grid service
          development kit that includes:
             Primitives, APIs, tools and runtime services
              designed to ease the task of building OGSI-
              compliant services
             Primitives for provisioning security
             Base services that provide an infrastructure
              with which to build higher-level services




Open Grid Services Architecture                              97
        GT3 User-Defined Services (cont.)

                                        GT3 build files

 User source files
                                                          Grid
                                                          service
                                  ANT                     executable
                                                          files
   (Diagram inspired by
   Borja Sotomayor’s
   excellent GT3 tutorial)              User build file


Open Grid Services Architecture                                   98
                                  Overview
         Grid background
         Open Grid Services Architecture
         Open Grid Services Infrastructure
         Beyond OGSI: other OGSA services
         Globus Toolkit v3 implementation
         Early GT3 performance results
         Scientific and commercial perspectives
         Summary


Open Grid Services Architecture                    99
          Early GT3 Performance Analysis
         Goal: explore GT3 under heavy load
             Maximal throughput/rate of GT3 services
             Identify limiting factors
         Three GT3 Grid services measured
             GRAM
             DummyService
             IndexService
         Note: Preliminary results, performance
          continues to improve rapidly
         Work performed by Massimo Lamanna,
          David Foster, et al., at CERN
Open Grid Services Architecture                         100
          GT3 GRAM Performance: Setup
         GRAM in GT3 standalone container
         Managed-job-globusrun clients started
          simultaneously on up to 32 client nodes
          (lxplus) in non-batch mode used to submit
          jobs to GT3 GRAM
         GRAM hardware: 2 * Intel Pentium III
          600MHz processors, 256MB RAM
         Note: 1 managed-job-globusrun client is
          able to submit 1 job


Open Grid Services Architecture                       101
                 GT3 GRAM performance
         Results: service node
             Saturation throughput for job submission on
              the service node: 3.8 jobs/minute with an
              average CPU user+system usage of 62%
          



          




         Comments:
             Scalability issue for heavily used servers
Open Grid Services Architecture                             102
                 GT3 GRAM performance
         Results: client node
             Using a 2 * Intel Pentium III 600MHz
              processors, 256MB RAM client node, a
              managed-job-globusrun client consumes at
              average 16 seconds CPU user+system time
              (on both CPU’s) for 1 job
         Comment:
             Lightweight clients (e.g., written in "C")
              needed



Open Grid Services Architecture                            103
                DummyService performance
      Setup (1)
          Each DummyService client executes these steps:
           1.   Calls DummyServiceFactory to create a DummyService instance
           2.   Executes 2 simple methods (echo and getTime) on the
                DummyService instance
           3.   Calls DummyService instance to destroy itself
          Up to 1000 clients talking to the DummyService
           from up to 45 client nodes (lxplus)
          With & without GSI message level authentication
          Grid service node hardware: 2 * Intel Pentium III
           600MHz processors, 256MB RAM
      Setup (2)
          Same, but repeats step #2 one hundred times
Open Grid Services Architecture                                        104
             DummyService performance
   Preliminary Results
     setup   authen-        service          saturation        average CPU
             tication      container         throughput       u+s usage, %

      1         -       GT3 standalone     41 services/s          89

      1        yes      GT3 standalone     1.3 services/s         88

      1         -           Tomcat         60 services/s          89
      1        yes          Tomcat         1.2 services/s         88
      2         -       GT3 standalone   300 method calls/s       96
      2        yes      GT3 standalone   10 method calls/s        72
      2         -           Tomcat       290 method calls/s       96
      2        Yes          Tomcat       13 method calls/s        79

   Security overhead needs further investigation
   Cross check our implementation/setup with Globus team foreseen

Open Grid Services Architecture                                              105
              DummyService Performance
         Conclusions
             Security overhead needs further investigation
             More tests on more powerful machines
             Container: depending on the setup the Tomcat
              container may be a bit slower or up to 50%
              faster, compared to the standalone container
         Notes:
             Results table shows top saturation rates
             With varying number of clients throughput
              goes down by up to 30% and the average CPU
              u+s usage varies accordingly
Open Grid Services Architecture                               106
Open Grid Services Architecture   107
         The first time the client contacts the
          DummyServiceFactory, creates a
          DummyService instance, and calls the first
          method, it takes about 10s to accomplish it
         These actions take about 1s on subsequent
          occasions
Open Grid Services Architecture                         108
               IndexService performance
         Setup (1): IndexService acting as a
          notification source (pushing data)
             Multiple notification sinks subscribe to the
              IndexService "Host" Service Data Element (SDE),
              and are notified about each update of "Host" SDE,
              happening at a fixed rate
             No security
             Grid service node hardware: 2 * Intel Pentium III
              600MHz processors, 256MB RAM
         Setup (2): IndexService responding to
          findServiceData requests (pulling data)
             Multiple ogsi-find-service-data clients are run
              sequentially and in parallel asking for IndexService
              "Host" Service Data Element
             No security
             Grid service node hardware: 2 * Intel Pentium III
              600MHz processors, 256MB RAM
Open Grid Services Architecture                                      109
                IndexService Performance

        setup         service           saturation         average CPU
                     container         throughput              u+s
                                                            usage, %
          1       GT3 standalone   10-15 notifications/s    81 – 87

          1           Tomcat                -

          2       GT3 standalone     200 requests/s            88

          2           Tomcat         200 requests/s            90




Open Grid Services Architecture                                          110
                 IndexService Performance
      Saturation throughput with findServiceData is
       ~13-20 times higher than with notifications
      Setup (1) measurement using Tomcat failed
       due to a thread bug, is fixed, fix announced to
       appear in next (4.1.28) Tomcat version
      Preliminary measurement with a faster service
       node (2 * Intel(R) Xeon(TM) 2.40GHz
       processors, 1GB RAM):
           Saturation throughput for setup (1) was about
            32 notifications/s for 800 listeners compared to
            10 notifications/s with 400 listeners on the 2 *
            600MHz machines – not quite 4 times faster
Open Grid Services Architecture                                111
Open Grid Services Architecture   112
Open Grid Services Architecture   113
                                  Overview
         Grid background
         Open Grid Services Architecture
         Open Grid Services Infrastructure
         Beyond OGSI: other OGSA services
         Globus Toolkit v3 implementation
         Early GT3 performance results
         Scientific and commercial perspectives
         Summary


Open Grid Services Architecture                    114

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:7
posted:11/11/2011
language:English
pages:114