Docstoc

EScience2007.ppt - Globus - Globus Project

Document Sample
EScience2007.ppt - Globus - Globus Project Powered By Docstoc
					Grid Computing and Globus
        Toolkit 4

   Rachana Ananthakrishnan
  Argonne National Laboratory
               What is a Grid?

l   Resource sharing
    – Computers, storage, sensors, networks, …
    – Sharing always conditional: issues of trust,
      policy, negotiation, payment, …
l   Coordinated problem solving
    – Beyond client-server: distributed data
      analysis, computation, collaboration, …
l   Dynamic, multi-institutional virtual orgs
    – Community overlays on classic org structures
    – Large or small, static or dynamic

                                                     2
               An Old Idea …

l   “The time-sharing computer system can unite
    a group of investigators …. one can conceive
    of such a facility as an … intellectual public
    utility.”
    – Fernando Corbato and Robert Fano, 1966
l   “We will perhaps see the spread of ‘computer
    utilities’, which, like present electric and
    telephone utilities, will service individual
    homes and offices across the country.”
    – Len Kleinrock, 1967

                                                 3
     Why Is this Hard or Different?

l   Lack of central control
    – Where things run
    – When they run
l   Shared resources
    – Contention, variability
l   Communication and coordination
    – Different sites implies different sys admins,
      users, institutional goals, and often socio-
      political constraints



                                                      4
          Why Grid? —
   The Changing Nature of Work

                                 Project focused, globally distributed
                                 Project focused, globally distributed
  Collaborative & Dynamic
  Collaborative & Dynamic       teams, spanning organizations within
                                teams, spanning organizations within
                                  and beyond company boundaries
                                  and beyond company boundaries

                                Each team member/group brings own
                                Each team member/group brings own
Distributed & Heterogeneous
Distributed & Heterogeneous     data, compute, & other resources into
                                data, compute, & other resources into
                                            the project
                                             the project

                                    Access to computing and data
                                    Access to computing and data
Data & Computation Intensive
Data & Computation Intensive    resources must be coordinated across
                                resources must be coordinated across
                                          the collaboration
                                          the collaboration

                                   Resources must be available to
                                    Resources must be available to
Concurrent Innovation Cycles
Concurrent Innovation Cycles       projects with strong QoS, & also
                                   projects with strong QoS, & also
                                 reflect enterprise-wide biz priorities
                                 reflect enterprise-wide biz priorities


            IT must adapt to this new reality
            IT must adapt to this new reality
                                                                     5
               For Example:
             Digital Astronomy
l   Digital observatories provide online
    archives of data at different wavelengths




l   Ask questions such as: what objects are
    visible in infrared but not visible spectrum?



                                                    6
 For Example:
Cancer Biology




                 7
         System-Level                 Decomposition
           Problem




                 Implementation
Facilities   U. Colorado
Computers                                          UIUC
              Experimental                   Experimental
Storage          Model                          Model
Networks                          COORD.

Services
Software                           NCSA
People                       Computational Model
                                                            8
                    DOE Earth System Grid

Goal: Enable
sharing &
analysis of
high-volume
data from
advanced
earth system
models



www.earthsystemgrid.org                     9
        What Kinds of Applications?
l   Computation intensive
    – Interactive simulation (climate modeling)
    – Large-scale simulation and analysis (galaxy
      formation, gravity waves, event simulation)
    – Engineering (parameter studies, linked models)
l   Data intensive
    – Experimental data analysis (e.g., physics)
    – Image & sensor analysis (astronomy, climate)
l   Distributed collaboration
    – Online instrumentation (microscopes, x-ray)
      Remote visualization (climate studies, biology)
    – Engineering (large-scale structural testing)      10
      Underlying Problem:
The Application-Infrastructure Gap


              Dynamic
               and/or
             Distributed
             Applications
   Shared Distributed Infrastructure
                             A           B


                         1           1



                                 9           9
                                             11
                 Bridging the
          Application-Resource Gap

                             User
                           Application
                                                  Tool
          Tool
                                    Workflow
                                               User Svc
 Uniform interfaces,                           Host Env
security mechanisms,     Registry
Web service transport,                              Credent.
     monitoring
                                                     Data
        Execution     User Svc          Data         Mgmt
         Mgmt.       Host Env          Transfer

                                                     Database
                    Specialized
     Computers                         Storage             12
                     resource
            Grid Infrastructure
l   Distributed management
    – Of physical resources
    – Of software services
    – Of communities and their policies
l   Unified treatment
    – Build on Web services framework
    – Use standards (WSRF/WS-Transfer/Man) to
      represent/access state
    – Common management abstractions &
      interfaces


                                                13
More Specifically, I May Want To …
l   Create a service for use by my colleagues
l   Manage who is allowed to access my
    service (or my experimental data or …)
l   Ensure reliable & secure distribution of
    data from my lab to my partners
l   Run 10,000 jobs on whatever computers I
    can get hold of
l   Monitor the status of the different
    resources to which I have access


                                                14
  The Globus Approach:
Philosophy and Technology
                  Globus is…

l   A collection of tools to solve problems that
    come up frequently when building
    collaborative distributed applications
l   Software for Grid infrastructure
    – Service enable new & existing resources
    – Uniform abstractions & mechanisms
l   Tools to build applications that exploit Grid
    infrastructure
    – Registries, security, data management, …
l   Open source & open standards
    – Each empowers the other
l   Enabler of a rich tool & service ecosystem
                                                    16
              Globus is an Hour Glass

l   Local sites have their own            Higher-Level Services
                                               and Users
    policies, installs – heterogeneity!
    – Queuing systems, monitors,
      network protocols, etc    Standard
l   Globus unifies – standards! Interfaces
    – Build on Web services
    – Use WS-RF, WS-Notification to
      represent/access state
    – Common management
                                            Local heterogeneity
      abstractions & interfaces
                                                             17
        Globus is a Building Block
l   Basic components for Grid functionality
    – Not turnkey solutions, but building blocks &
      tools for application developers & system
      integrators
l   Highest-level services are often application
    specific, we let applications concentrate
    there
l   Easier to reuse than to reinvent
    – Compatibility with other Grid systems
      comes for free
l   We provide basic infrastructure to get you
    one step closer                                  18
             Globus Philosophy
l   Globus was first established as an open
    source project in 1996
l   The Globus Toolkit is open source to:
    – Allow for inspection
       > for consideration in standardization processes

    – Encourage adoption
       > in pursuit of ubiquity and interoperability

    – Encourage contributions
       > harness the expertise of the community

l   The Globus Toolkit is distributed under the
    (BSD-style) Apache License version 2
                                                          19
                  dev.globus

l   Governance model based on Apache Jakarta
    – Consensus based decision making
l   Globus software is organized as several
    dozen “Globus Projects”
    – Each project has its own “Committers”
      responsible for their products
    – Cross-project coordination through shared
      interactions and committers meetings
l   A “Globus Management Committee”
    – Overall guidance and conflict resolution

                                                  20
                  http://dev.globus.org


 Guidelines
  (Apache
  Jakarta)

Infrastructure
 (CVS, email,
bugzilla, Wiki,
   licenses)



   Projects
   Include
      …




                                          21
        Globus Technology Areas

l   Core runtime
    – Infrastructure for building new services
l   Security
    – Apply uniform policy across distinct systems
l   Execution management
    – Provision, deploy, & manage services
l   Data management
    – Discover, transfer, & access large data
l   Monitoring
    – Discover & monitor dynamic services
                                                     22
        Non-Technology Projects

l   Distribution Projects
    – Globus Toolkit Distribution
    – Process in use since April ’07
l   Documentation Projects
    – GT Release Manuals
l   Incubation Projects
    – Incubation management project
    – And any new projects wanting to join




                                             23
           Globus Software: dev.globus.org
    Globus Projects                                          Globus
                                                   GRAM
                                                             Toolkit
   MPICH G2       Java                              Data       Replica
                           Delegation   MyProxy
                 Runtime                            Rep       Location

   OGSA-DAI        C                      GSI-
                              CAS                  GridFTP     MDS4
                 Runtime                OpenSSH

   Incubation                                     Reliable
     Mgmt        Python
                             C Sec      GridWay     File     GT4 Docs
                 Runtime
                                                  Transfer


 Incubator
 Projects


Common                Execution                     Info
Runtime
           Security
                        Mgmt
                                     Data Mgmt
                                                  Services
                                                                Other    24
              Globus Software: dev.globus.org
     Globus Projects                                                   Globus
                                                                GRAM
                                                                       Toolkit
    MPICH G2            Java                                    Data         Replica
                                 Delegation    MyProxy
                       Runtime                                  Rep         Location

     OGSA-DAI            C                       GSI-
                                    CAS                     GridFTP          MDS4
                       Runtime                 OpenSSH

    Incubation                                              Reliable
      Mgmt             Python
                                   C Sec       GridWay        File       GT4 Docs
                       Runtime
                                                            Transfer


 Incubator                       Swift     GEMLCA        RAVI      Falkon      MonMan
 Projects              GAARDS MEDICUS Cog WF Virt WkSp             GARS       NetLogger

 GDTE       GridShib   OGRO      UGP       Dyn Acct Gavia JSC      DDM         Metrics
Introduce   PURSE      HOC-SA    LRMA         WEEP   Gavia MS      SGGC       ServMark

Common                     Execution                         Info
Runtime
               Security
                             Mgmt
                                           Data Mgmt
                                                           Services
                                                                              Other    25
    Incubator Process in dev.globus

l   Entry point for new Globus projects
l   Incubator Management Project (IMP)
    – Oversees incubator process form first
      contact to becoming a Globus project
    – Quarterly reviews of current projects
http://dev.globus.org/wiki/Incubator/
  Incubator_Process




                                              26
              Globus Software: dev.globus.org
     Globus Projects                                                   Globus
                                                                GRAM
                                                                       Toolkit
    MPICH G2            Java                                    Data         Replica
                                 Delegation    MyProxy
                       Runtime                                  Rep         Location

     OGSA-DAI            C                       GSI-
                                    CAS                     GridFTP          MDS4
                       Runtime                 OpenSSH

    Incubation                                              Reliable
      Mgmt             Python
                                   C Sec       GridWay        File       GT4 Docs
                       Runtime
                                                            Transfer


 Incubator                       Swift     GEMLCA        RAVI      Falkon      MonMan
 Projects              GAARDS MEDICUS Cog WF Virt WkSp             GARS       NetLogger

 GDTE       GridShib   OGRO       UGP      Dyn Acct Gavia JSC      DDM         Metrics
Introduce   PURSE      HOC-SA    LRMA         WEEP   Gavia MS      SGGC       ServMark

Common                     Execution                         Info
Runtime
               Security
                             Mgmt
                                           Data Mgmt
                                                           Services
                                                                              Other    27
              Globus Software: dev.globus.org
     Globus Projects                                                   Globus
                                                                GRAM
                                                                       Toolkit
    MPICH G2            Java                                    Data         Replica
                                 Delegation    MyProxy
                       Runtime                                  Rep         Location

     OGSA-DAI            C                       GSI-
                                    CAS                     GridFTP          MDS4
                       Runtime                 OpenSSH

    Incubation                                              Reliable
      Mgmt             Python
                                   C Sec       GridWay        File       GT4 Docs
                       Runtime
                                                            Transfer


 Incubator                       Swift     GEMLCA        RAVI      Falkon      MonMan
 Projects              GAARDS MEDICUS Cog WF Virt WkSp             GARS       NetLogger

 GDTE       GridShib   OGRO       UGP      Dyn Acct Gavia JSC      DDM         Metrics
Introduce   PURSE      HOC-SA    LRMA         WEEP   Gavia MS      SGGC       ServMark

Common                     Execution                         Info
Runtime
               Security
                             Mgmt
                                           Data Mgmt
                                                           Services
                                                                              Other    28
         Globus User Community

l   Large & diverse
    – 10s of national Grids, 100s of applications,
      1000s of users; probably much more
    – Every continent except Antarctica
    – Applications ranging across many sciences
    – Dozens (at least) of commercial deployments
l   Successful
    – Many production systems doing real work
    – Many applications producing real results
l   Smart, energetic, demanding
    – Constant stream of new use cases & tools       29
  Global
Community




            30
              Examples of
        Production Scientific Grids
l   APAC (Australia)
l   China Grid
l   China National Grid
l   DGrid (Germany)
l   EGEE
l   NAREGI (Japan)
l   Open Science Grid
l   Taiwan Grid
l   TeraGrid
l   ThaiGrid
l   UK Nat’l Grid Service

                                      31
More Specifically, I May Want To …
l   Create a service for use by my colleagues
l   Manage who is allowed to access my service
    (or my experimental data or …)
l   Ensure reliable & secure distribution of data
    from my lab to my partners
l   Run 10,000 jobs on whatever computers I can
    get hold of
l   Monitor the status of the different resources to
    which I have access


                                                       32
                                 Web Service

      l   Web Services are basic distributed
          computing technology that let us construct
          client-server interactions




Borja Sotomayor , http://gdp.globus.org/gt4-tutorial/multiplehtml/ch01s02.html   33
                Web Services

l   Web services
    – Are platform independent
    – Are language independent
    – Describe themselves
    – Are ideal for loosely coupled systems
l   Standards for defining & accessing services
    – WSDL: Web Services Description Language
    – SOAP: Simple Object Access Protocol
    – Also other standards for security, state
      access, etc., etc.

                                                  34
                   WSDL: Web Services
                   Description Language



                                     Define expected messages for a service,
                                     and their (input or output parameters)


                                     An interface groups together a number of
                                     messages (operations)




Bind an Interface via a
definition to a specific transport         The network location where the service is
(e.g. HTTP) and messaging                  implemented , e.g. http://localhost:8080
(e.g. SOAP) protocol
                                                                                35
       Web Services:
E.g., File Transfer Service
              User
                                    WSDL
                                   defining
                                   “Move”
          Move F from
                                  operation,
            A to B
                                     Etc.


             Interface



        Implementation


    Hosting environment/runtime
         (“C”, Axis, .NET, …)
                                               36
    “Stateless” vs. “Stateful” Services

    FileTransfer
       Service
                                move (A to B)       Client
               move




l    Without state, how does client:
       –   Determine what happened (success/failure)?
       –   Find out how many files completed?
       –   Receive updates when interesting events arise?
       –   Terminate a request?
l    Few useful services are truly “stateless”, but
     WS interfaces alone do not provide built-in
     support for state
                                                             37
                  FileTransferService
                    (without WSRF)

    FileTransfer
       Service
                         move (A to B) : transferID   Client
               move
          whatHappen
    state tellMeWhen
             cancel


l    Developer reinvents wheel for each new service
       – Custom management and identification of state:
         transferID
       – Custom operations to inspect state synchronously
         (whatHappen) and asynchronously (tellMeWhen)
       – Custom lifetime operation (cancel)

                                                               38
                  WSRF in a Nutshell

                                 l   Service
                                 l   State representation
                                     – Resource
        Service
                     GetRP           – Resource Property
EPR                GetMultRPs    l   State identification
 EPR                                 – Endpoint Reference
  EPR                SetRP
        Resource                 l   State Interfaces
                    QueryRPs
                                     – GetRP, QueryRPs,
          RPs       Subscribe          GetMultipleRPs, SetRP
                   SetTermTime   l   Lifetime Interfaces
                                     – SetTerminationTime
                     Destroy
                                     – ImmediateDestruction
                                 l   Notification Interfaces
                                     – Subscribe
                                     – Notify
                                 l   ServiceGroups
                                                               39
       FileTransferService (w/ WSRF)

FileTransferService
                                                             Client
            createResource   createResource (A to B) : EPR

    Transfer    getRP

      RPs      queryRPs

               destroy

l     Developer specifies custom method to createResource
      and leaves the rest to WSRF standards:
       – State exposed as Resource + Resource Properties and
         identified by Endpoint Reference (EPR)
       – State inspected by standard interfaces (GetRP, QueryRPs)
       – Lifetime management by standard interfaces (Destroy)




                                                                      40
         Globus Toolkit:
  Open Source Grid Infrastructure
                   Data                      Globus Toolkit v4
                 Replication                  www.globus.org
                                    Grid
 Credential         Replica
                                 Telecontrol
   Mgmt            Location
                                   Protocol
                                 Community
                 Data Access                                Python
 Delegation                      Scheduling      WebMDS
                 & Integration                              Runtime
                                 Framework
                   Reliable
 Community                        Workspace                   C
                     File                        Trigger
Authorization                    Management                 Runtime
                   Transfer
                                 Grid Resource
Authentication                                               Java
                   GridFTP        Allocation &    Index
Authorization
                                  Management                Runtime

                    Data         Execution         Info     Common
 Security
                    Mgmt           Mgmt          Services   Runtime
                                                                      41
                          GT4 and Web Services


                                      User Applications



                                    Custom          GT4
   (e.g., Apache Axis)




                                                                  and Admin
                                                 WSRF Web
     GT4 Container




                                                                   Registry
                         Custom      WSRF
                          Web       Services      Services
                         Services

                                    WS-A, WSRF, WS-Notification


                                    WSDL, SOAP, WS-Security


Java (standard Apache Axis), C (fast, small footprint), Python 42
         GT4 WS Core in a Nutshell

                                 Implementation of WSRF:
                                       Resources,
                                   EndpointReferences,
                                   ResourceProperties
        Service
                     GetRP                         Operation Providers: pre-
                                                   build implementations of
EPR                GetMultRPs                          WSRF operations
 EPR
  EPR                SetRP
        Resource                                Notification implementation:
                    QueryRPs                    Topics, TopicSet, Embedded
          RPs                                  Notification Consumer service
                    Subscribe

                   SetTermTime             Implementations of Resources
                                                (ReflectionResource,
                     Destroy
                                           PersistentReflectionResource)
                                              and ResourceProperties
                                             (SimpleResourceProperty,
                                            ReflectionResourceProperty)



                                                                               43
         GT4 WS Core in a Nutshell


                                         ResourceHome: The home
        Service                            “owns” the Resource
                     GetRP                instances in the service

EPR                GetMultRPs                       SingletonResourceHome:
 EPR                                                manages single instance
  EPR                SetRP                                 of Resource
        Resource
                    QueryRPs
          RPs                                       ServiceResourceHome:
                    Subscribe
                                                   for services that support
                   SetTermTime                         a single Resource
                                                            instance
  ResourceHome       Destroy     ResourceHomeImpl: manages
                                 multiple Resource instances.
                                  Supports resources with in-
                                 memory state and resources
                                 with persistent (on disk) state



                                                                               44
             GT4 WS Core in a Nutshell


Service Container
                                        Service Container: host
            Service                       multiple services in
             Service      GetRP           container; one JVM
               Service      GetRP               process
                             GetRP
                       GetMultRPs
   EPR                   GetMultRPs
    EPR                    GetMultRPs
      EPR
     EPR                  SetRP         …more details: based
        EPR
       EPRResource          SetRP
         EPRResource          SetRP        on AXIS service
              Resource QueryRPs
                          QueryRPs      container, processes
              RPs           QueryRPs
                        Subscribe         SOAP messages,
               RPs        Subscribe       ResourceContext
                 RPs        Subscribe
                      SetTermTime            extension.
                        SetTermTime
       ResourceHome       SetTermTime
                         Destroy
         ResourceHome      Destroy
          ResourceHome       Destroy




                                                                  45
             GT4 WS Core in a Nutshell


Service Container                                  Secure Communication:
                                                    Transport, Message,
            Service                     PIP
                                                   Conversation (Transport
             Service      GetRP                      demonstrates best
               Service      GetRP
                             GetRP      PDP            performance)
   EPR                 GetMultRPs
    EPR                  GetMultRPs
     EPR
      EPR                  GetMultRPs
                          SetRP
        EPR
       EPRResource          SetRP
         EPRResource          SetRP                  Configurable Security
              Resource QueryRPs
                          QueryRPs                Policies: Policy Information
              RPs           QueryRPs
                        Subscribe                Points (PIPs), Policy Decision
               RPs        Subscribe
                 RPs        Subscribe               Points (PDP) -- chained
                      SetTermTime
                        SetTermTime
       ResourceHome       SetTermTime
                         Destroy
         ResourceHome      Destroy
          ResourceHome       Destroy          Example authorization
                                              PDPs: GridMap, SAML
                                                implementations,
                                                 XACML policies



                                                                                  46
             GT4 WS Core in a Nutshell


                                                WorkManager: “thread
Service Container
                                                pool”, site independent
                                         PIP       “work” manager
            Service
             Service      GetRP
               Service      GetRP
                             GetRP
                       GetMultRPs        PDP
   EPR                                                Apache Database
    EPR                  GetMultRPs
                           GetMultRPs              Connection Pool library
      EPR
     EPR                  SetRP                     (JDBC “DataSource”
        EPR
       EPRResource          SetRP
         EPRResource          SetRP                   implementation)
              Resource QueryRPs
                          QueryRPs
              RPs           QueryRPs
                        Subscribe
               RPs        Subscribe
                 RPs        Subscribe              JNDI Directory: manages
                      SetTermTime
                        SetTermTime                 internal, shared objects
       ResourceHome       SetTermTime
                         Destroy                       (ResourceHomes,
         ResourceHome      Destroy                       WorkManager,
          ResourceHome       Destroy               Configuration objects,…)

 WorkManager    DB Conn Pool   JNDI Directory



                                                                               47
                GT4 WS Core in a Nutshell

Apache Tomcat
                                                     Deploy Service
  Service Container
                                                  Container “standalone”
                                           PIP      or within Apache
              Service                                    Tomcat
               Service      GetRP
                 Service      GetRP
                               GetRP
                         GetMultRPs        PDP
     EPR                   GetMultRPs
      EPR                    GetMultRPs
        EPR
       EPR                  SetRP
          EPR
         EPRResource          SetRP
           EPRResource          SetRP
                Resource QueryRPs
                            QueryRPs
                RPs           QueryRPs
                          Subscribe
                 RPs        Subscribe
                   RPs        Subscribe
                        SetTermTime
                          SetTermTime
         ResourceHome       SetTermTime
                           Destroy
           ResourceHome      Destroy
            ResourceHome       Destroy

   WorkManager    DB Conn Pool   JNDI Directory



                                                                           48
           The Introduce Authoring Tool

l    Define service
l    Create skeleton
l    Discover types
l    Add operations
l    Configure security
l    Modify service

    See also: SOAPLab,
    OPAL, pyGlobus,
    Gannon, etc.

        Introduce: Hastings, Saltz, et al., Ohio State University   49
                Generated Service Features
       l   Dynamic discovery and use of published data types
       l   Creates WSDL2.0 / WSRF compliant services
       l   Supports creating multiple resource/services using
           the Web Service Resource Framework (WSRF)
       l   Globus GSI Security Configuration
       l   Authorization Support
       l   Resource Property configuration and monitoring
           service registration
       l   Rich extension/plug-in framework for creating
           custom services or adding custom functionality to
           Introduce


Shannon Hastings, http://dev.globus.org/wiki/Incubator/Introduce   50
Shannon Hastings, http://dev.globus.org/wiki/Incubator/Introduce   51
                Generated Service Skeleton




Shannon Hastings, http://dev.globus.org/wiki/Incubator/Introduce   52
                Generated Service Skeleton

                                                                   = introduce generated
                                                                   = globus/axis generated
                                                                   = developers contribution




Shannon Hastings, http://dev.globus.org/wiki/Incubator/Introduce                      53
More Specifically, I May Want To …

l   Create a service for use by my colleagues
l   Manage who is allowed to access my
    service (or my experimental data or …)
l   Ensure reliable & secure distribution of
    data from my lab to my partners
l   Run 10,000 jobs on whatever computers I
    can get hold of
l   Monitor the status of the different
    resources to which I have access

                                                54
                Security Basics

l   Privacy
    – Only the sender and receiver should be able
      to understand the conversation
l   Integrity
    – Receiving end must know that the received
      message was not altered
l   Authentication
    – Prove that user is who he claims to be
l   Authorization
    – Is user allowed to perform the action

                                                    55
          Grid Security Concerns

l   Control access to shared services
    – Address autonomous management, e.g.,
      different policy in different work groups
l   Support multi-user collaborations
    – Federate through mutually trusted services
    – Local policy authorities rule
l   Allow users and application communities to
    set up dynamic trust domains
    – Personal/VO collection of resources working
      together based on trust of user/VO

                                                    56
Virtual Organization (VO) Concept




 l   VO for each application or workload
 l   Carve out and configure resources for a
     particular use and set of users
                                               57
               Globus Security

l   Globus security is based on the Grid
    Security Infrastructure (GSI)
    – Set of IETF standards for security
      interaction
l   Public-key-based authentication using
    X509 certificates
l   Additionally provides
    – proxy certificates
    – delegation


                                            58
    Message Security & Authentication


         Private key is             Public key is
         known only to              given away to
         owner                      the world


l    Encrypt with one decrypt with other
l    Digital signature used for message
     integrity
l    Encryption used for message privacy
l    Authentication protocols leverage keys


                                                    59
    Public Key Infrastructure (PKI)
l   PKI allows you to know
    that a given public key
    belongs to a given user
l   PKI builds off of
    asymmetric encryption:
    – Each entity has two             r
      keys: public and private    w ne
                                 O
    – The private key is known
      only to the entity
l   The public key is given to
    the world encapsulated
    in a X.509 certificate




                                          60
                          Authentication Using
                           Digital Certificates


      l   Digital document that certifies a public key
          is owned by a particular user
      l   Signed by 3rd party – the Certificate
          Authority (CA)




Borja Sotomayor , http://gdp.globus.org/gt4-tutorial/multiplehtml/ch09s04.html   61
                          Authentication Using
                           Digital Certificates


      l   Digital document that certifies a public key
          is owned by a particular user
      l   Signed by 3rd party – the Certificate
          Authority (CA)
                                        To know if you
                                        should trust the
                                        certificate, you
                                        just have to trust
                                        the CA



Borja Sotomayor , http://gdp.globus.org/gt4-tutorial/multiplehtml/ch09s04.html   62
                   Certificates
l   Similar to passport or driver’s license




     Name                       John Doe
     Issuer                     755 E. Woodlawn      State of
                                                     Illinois
     Public Key                 Urbana IL 61801        Seal
     Validity                    BD 08-06-65
     Signature                   Male 6’0” 200lbs
                                 GRN Eyes        Valid Till: 01-02-2008




                                                                          63
         Requesting a Certificate
l   To request a
    certificate a user
    starts by generating
    a key pair




                           Private Key   Public Key

                                                      64
             Certificate Request
l   The user signs their
    own public key to              Public Key
    form what is called a
    Certificate Request
l   Email/Web upload
l   Note private key is         Sign
    never sent anywhere

                            Certificate
                             Request

                            Public Key

                                                65
       Registration Authority (RA)
l   The user then takes
    the certificate to a
    Registration
    Authority (RA)
l   Vetting of user’s
    identity
l   Often the RA
    coexists with the CA
    and is not apparent    Certificate
    to the user             Request
                                         ID
                           Public Key

                                              66
            Certificate Issuance
l   The CA then takes
                                                 Certificate
    the identity from the                         Request
    RA and the public
    key from the                             Public Key
    certificate request
                             Name
l   It then creates, signs
    and issues a
    certificate for the
    user
                                    Name
                                    Issuer
                                    Validity
                                    Public Key
                                    Signature                  67
          GSI: Proxy Credentials

l   Proxy credentials are short-lived
    credentials created by user
    – Proxy signed by certificate private key
l   Short term binding of user’s identity to
    alternate private key
l   Same effective identity as certificate


                  SIGN




                                                68
          GSI: Proxy Credentials

l   Stored unencrypted for easy repeated
    access
l   Chain of trust
    – Trust CA -> Trust User Certificate -> Trust
      Proxy
l   Key aspects:
    – Generate proxies with short lifetime
    – Set appropriate permissions on proxy file
    – Destroy when done



                                                    69
               GSI Delegation

l   Enabling another entity to run as you
l   Ensure
    – Limited lifetime
    – Limited capability




                                            70
Grid Security: Single Sign On
         Delegation




                                71
                          Delegation Service

   l   Higher level                        Hosting Environment

       service                  Service1

   l   Authentication           Service2
                                                       Resources

       protocol                            EPR   Delegation Service
       independent              Service3

                                                 Delegate    Refresh
   l   Refresh
       interface                                                   Refresh

   l   Delegate once,                            EPR
                                                        Delegate
       share across
       services and
       invocation                                       Client
Rachana Ananthakrishnan                                               72
          Globus Security Tools

l   Basic Grid Security Mechanisms
l   Certificate Generation Tools
l   Certificate Management Tools
    – Getting users “registered” to use a Grid
    – Getting Grid credentials to wherever they’re
      needed in the system
l   Authorization/Access Control Tools
    – Storing and providing access to system-
      wide authorization information



                                                     73
                   Simple CA

l   An online service that issues low-quality
    GSI certificates
l   Not a true Certificate Authority (CA)
    – No revoking or reissuing certificates
    – No verification of identities
    – The service itself is not especially secure
l   Most production Grids will not accept
    certificates that are not signed by a well-
    known CA


                                                    74
                    MyProxy
l   Service to store user
    credentials
l   Grid administrators pre-
    load credentials in the
    server for users to retrieve
l   Greatly simplifies
    certificate management
l   Online CA
l   Trust root provisioning


                                   75
           Portal-Based User Registration Service
                         (PURSE)
                         l   Portal extensions (CGI scripts) that
                             automate user registration requests
                             – Solicits basic data from user
                             – Generates cert request from CA
                             – Admin interface allows CA admin to
                               accept/reject request
                             – Generates a certificate and stores in
                               MyProxy service
                             – Gives user ID/password for MyProxy
Optional                 l   Benefits
 Review
                             – Users never have to deal with certs
                             – Portal can get user cert from MyProxy
                               when needed
                             – Database is populated with user data
                         l   Originally written for ESG, now
                             generalized                             76
           Globus Authorization

l   Extensible framework
l   C implementation
    – GSI callout
l   Java implementation
    – Pluggable policy decision points and policy
      information points




                                                    77
         Authorization Framework
l   Policy Information Points (PIPs)
    – Collect attributes (subject, action, resource)
l   Policy Decision Points (PDPs)
    – Evaluate authorization policy
l   Authorization Engine
    – Orchestrates authorization process
    – Enforce distributed authorization policy
    – Combining algorithm to render a decision



                                                       78
              GT 4.0 Authorization Framework
               Web Services Message Context (store attributes)




PIP1   PIP2   …      PIPn                 PDP1        PDP2       …   PDPn



                                                      Permit
                                      Permit

                                               Deny         Permit

                Authorization Engine
                  (Deny-override)



                                                    Deny
                                                   Permit


                                                    Policy
                                                 Enforcement
                                                     Point
                                                                            79
    AuthZ Framework Enhancements

l   Modular code base
l   Improved attribute processing
    – Normalized attribute representation
    – Comparison of attributes across sources
    – Merging of attributes of same entities
l   Pluggable combing algorithm
l   Targeted for GT 4.2




                                                80
     Authorization using a GridMap File

l   Maps distinguished names (found in
    certificates) to local names (such as login
    accounts)
    – schopf@mcs.anl.gov
    – jms@nesc.ed.ac.uk
    – u11270@sdsc.edu
l   Can also serve as a access control list for
    GSI enabled services
l   Can be a Policy Decision Point


                                                  81
                CAS:
    Community Authorization Service
l   Allows resource providers to
    specify
     – Course-grained access control
       policies in terms of communities
       as a whole
     – Fine-grained access control is
       delegated to the community
l   Resource providers maintain
    authority over their resources
    use
l   Can be used as a policy
    Decision Point


                                          82
  Globus Security: How It Works

                       Services




Compute
 Center

                                  Users


             VO

                                      83
  Globus Security: How It Works

                       Services




Compute
 Center

                                  Users
      Rights


               VO
                     Rights
                                      84
      Globus Security: How It Works

                                  Services




  Compute
   Center                  CAS


                                             Users
             Rights

Local policy
on VO identity        VO
or attribute
authority                        Rights
                                                 85
      Globus Security: How It Works

                                      Services (running
                                      on user’s behalf)

                      Access

  Compute                                                 Rights
   Center                      CAS


                                                     Users
             Rights

Local policy
on VO identity          VO
or attribute
authority                            Rights
                                                             86
        Globus Security: How It Works

Authz Callout                with Proxy
                             Certificates
                                             Services (running
                                             on user’s behalf)

                         Access

   Compute                                                       Rights
    Center                         CAS


                                                            Users
                Rights

Local policy
on VO identity             VO
or attribute
authority                                   Rights
                                                                    87
     Globus’s Use of
    Security Standards




Supported,   Supported,      Fastest,
 but slow    but insecure   so default
                                    88
More Specifically, I May Want To …
l   Create a service for use by my colleagues
l   Manage who is allowed to access my
    service (or my experimental data or …)
l   Ensure reliable & secure distribution of
    data from my lab to my partners
l   Run 10,000 jobs on whatever computers I
    can get hold of
l   Monitor the status of the different
    resources to which I have access


                                                89
             Data Management

l   Stage/move large data to/from nodes
    – GridFTP for basic file movement
    – Reliable File Transfer (RFT)
l   Replicate data
    – Publish replica
    – Data Placemen Service (under development)
l   Locate data of interest
    – Replica Location Service (RLS)



                                                  90
GridFTP
            GridFTP: The Protocol
l   Data transfer protocol that is
    –   High-performance
    –   Secure
    –   Reliable
    –   Optimized for high-bandwidth wide-area
        networks
l   GGF recommendation GFD.20




                                                 92
       GridFTP: Protocol Features

l   Basic Grid security
l   Multiple data channels for parallel transfers
l   Partial file transfers
l   Third-party transfers
l   Reusable data channels
l   Command pipelining




                                                    93
             GridFTP: The Service
l   Complete Globus code
    – No licensing issues
l   Stable code base
    – Over 1200 unique servers Jan ’06 – Jan ’07
    – Over 90 Million known transfers last year
l   IPv6 Support
l   Extensible
    – Transport: eXtensible Input/Output (XIO)
    – Pluggable front end and backend
l   Integrated instrumentation
l   Server side computation                        94
            Control and Data Channels
l   GridFTP (and FTP) use (at least) two separate socket
    connections:
     – A control channel for carrying the commands and
       responses
     – A data Channel for actually moving the data
l   Control Channel and Data Channel can be (optionally)
    completely separate processes.
      Typical            Separate                Striped
    Installation         Processes               Server
      Control              Control              Control
       Data
                            Data
                                                     Data
                                                            95
             Parallel Data Streams

l   Multiple TCP streams
    between sender and
    receiver
l   Sender pushes multiple
    blocks in parallel streams
l   Blocks reassembled at
    receiving side and put
    into correct order
l   Protection against
                                 Parallel Transfer
    dropped packets for each Fully utilizes bandwidth of
    stream                 network interface on single nodes
                                                        96
             Striped GridFTP Service

l   Multiple nodes work together as a single logical
    GridFTP server
l   Every node of the cluster is used to transfer
    data into/out of the cluster
    – Each node reads/writes
      only pieces they’re




                                                                                               Parallel Filesystem
                                        Parallel Filesystem
      responsible for
    – Head node coordinates transfers
l   Multiple levels of parallelism
    – CPU, bus, NIC, disk etc.
    – Maximizes use of Gbit+ WANs
                                                                   Striped Transfer
                                                                Fully utilizes bandwidth of
                                                              Gb+ WAN using multiple nodes.

                                                                                              97
      GridFTP Server Performance
l   TCP buffer size control
    – Tune buffers to latency of network
    – Regular FTP optimized for low latency networks, not
      tunable
l   Dramatic improvements for high latency WAN
    transfers
    – 90% of network utilization possible
    – 27 GB/s achieved with commodity hardware
l   Performance tuning using command line
    parameters
    – -p for number of parallel streams
    – -tcp-bs for TCP buffer size
    – -vb for performance feedback



                                                            98
Disk-to-disk on TeraGrid




                           99
    GridFTP: Pluggable Data Transport

l    XIO: eXtensible Input/Output
l    Library written in C
l    Provides a single API that supports
     multiple wire protocols
l    Standard Posix interfaces
     – open/close/read/write
l    Protocol implementations encapsulated as
     drivers



                                                100
Typical Approach (without XIO)

                                     Network
                                   Network
                                     Protocol
                 Protocol API     Network
                                   Protocol
                                  Protocol



  Application
                 POSIX IO
                                           Disk



                Proprietary API

                                  Special Device




                                                   101
        Globus XIO Approach

                                       Network
                                     Network
                                       Protocol
                           Driver   Network
                                     Protocol
                                    Protocol




              Globus XIO
Application                Driver
                                             Disk


                           Driver

                                    Special Device




                                                     102
      GridFTP: Pluggable Storage

l   Data Storage Interface (DSI)
    – Interfaces to various storage types
    – Implement simple functions such as send,
      receive, mkdir,…
    – DSI modules available for HPSS and SRB




                                                 103
            GridFTP: Sever Stack

l   Transport                             Example
    – Exactly one per stack              Driver Stack

    – Must be on the bottom              Compression

l   Transform
                                          Logging
    – Zero or many per stack
l   Control flows from user to the          TCP
    top of the stack, to the transport
    driver.




                                                        104
                GridFTP Client

l   Globus FTP client library (API):
    – Integration of data transport capabilities
      directly into applications
    – Plug-in architecture for installing fault
      recovery and performance tuning
      algorithms
    – Asynchronous programming model




                                                   105
            GridFTP: Tool Mechanics

l   Server mechanics
    – globus-gridftp-server
    – Usually runs as root
    – Usually run as a daemon; connections fork new
      process and setuid
    – Can run inetd/xinetd if so desired
    – Port 2811 is standard but is configurable
    – Logging and security highly configurable
l   Client mechanics
    – globus-url-copy
    – Options for parallel channels, TCP buffer size, data
      buffer size, debugging, recursive directory transfers,
      etc.

                                                               106
            Recent Improvements:
              GridFTP over SSH

l   The Problem
    – Not all users require GSI
      and the need for
      certificate infrastructure.
l   The Solution
    – Use SSH for Control
      Channel
    – Data channel remains as
      is, so performance is still
      GridFTP
l   Included in 4.1.2
    development release
                                    107
Reliable File Transfer Service
       RFT - File Transfer Queuing

l   A WSRF service for queuing file transfer
    requests
    – Server-to-server transfers
    – Check pointing for restarts
    – Database back-end for failovers
l   Allows clients to requests transfers and
    then “disappear”
    – No need to manage the transfer
    – Status monitoring available if desired



                                               109
     Reliable File Transfer (RFT)

                          RFT Client
         Web Service
           invocation                        Optional
      (SOAP via https)                       notifications
                                                             Relational
                                                             Relational
                                                             Database
                                                             Database
                         RFT Service                         preserves
                                                             preserves
                                                               state
                                                               state
                                               Client API
                                               speaks GridFTP
                                               protocol


                         Multiple parallel
   GridFTP Control        data channels
                                              GridFTP Control
                           move files
     GridFTP Data                              GridFTP Data


Has transferred >900,000 files.                                           110
           Globus RFT Features

l   Supports concurrency
l   Restart markers saved by service in DB
    – Failed transfers restarted from midpoint
    – Configurable number of retries
    – Configurable exponential back off
    – Transfer all or none
l   Clients check status in two ways
    – Subscribe to notifications from RFT service
    – Poll service to find status of transfers


                                                    111
            Globus RFT Interfaces

l   Service exposes WSRF compliant interface
l   Single RFT service fronts multiple RFT
    resources
    – Each “user” can have separate resource
    – Each resource maintains own queue, notifications,
      lifetime
l   Operations:
    –   Create transfer
    –   Start transfer
    –   Get status
    –   Delete sets of files/directories
    –   WSRF complaint operations


                                                          112
            RFT: Tool Mechanics

l   RFT Service
    – Runs in Globus Java WS container/Tomcat
    – Uses JDBC capable database; PostgreSQL
      and MySQL most widely tested and used
l   RFT clients
    – rft and rft-delete: simple clients, not
      intended for production use
    – Recommend application-specific Web
      Services clients developed against the
      service WSDL


                                                113
                So we can…

l   Move files between servers
l   Reliably
l   With or without a Web service interface


l   But we wanted to work with replicas!




                                              114
Replica Location Service
     Globus Replica Location Service

l   Maintains mappings between logical identifiers
    and target names
l   Logical identifier or Logical File Name (LFN)
    – Location-independent identifier (name)
    – Example: foo
l   Target name or Physical File Name (PFN)
    – Specific file identifier such as a URL
    – E.g.: gsiftp://myserver.mycompany.com/foo
l   RLS maps between LFNs and PFNs
    – foo Þ gsiftp://myserver.mycompany.com/foo
                                                    116
                            LFNs and PFNs


 l   LFN to PFN mappings are often many-to-one
 l   Multiple PFNs may indicate different access to
     a file

access via GridFTP server
access via GridFTP server

access via one NFS mount
access via one NFS mount
                              foo Þ gsiftp://dataserver.mycompany.com/foo
                             foo Þ gsiftp://dataserver.mycompany.com/foo
                              foo Þ file://nodeA.mycompany.com/foo
                             foo Þ file://nodeA.mycompany.com/foo
access via 2nd NFS mount
access via 2nd NFS mount      foo Þ file://nodeB.mycompany.com/foo
                             foo Þ file://nodeB.mycompany.com/foo
                              foo Þ https://www.mycompany.com/foo
                             foo Þ https://www.mycompany.com/foo
access via web server
access via web server




                                                                            117
            Local Replica Catalog
   l   Local replica catalog (LRC): Catalog of
       LFN to PFN mappings
   l   LRCs contain consistent information
       about local to target mappings


           Local Replica Catalog (LRC)
            Local Replica Catalog (LRC)
fee Þ gsiftp://dataserver.mycompany.com/fee
 fee Þ gsiftp://dataserver.mycompany.com/fee
fii Þ file://nodeA.mycompany.com/fii
 fii Þ file://nodeA.mycompany.com/fii
foo Þ file://nodeB.mycompany.com/foo
 foo Þ file://nodeB.mycompany.com/foo
fum Þ https://www.mycompany.com/fum
 fum Þ https://www.mycompany.com/fum
                                                 118
          Replica Location Index

l   Replica Location Index (RLI): Aggregate
    information about one or more LRCs
l   Only the LFN content for LRC is aggregated
    – Each configured LRC sends list of LFNs to
      LRCs
    – PFNs and mappings not aggregated

                   RLI               RLI




           LRC     LRC     LRC    LRC      LRC

                                                  119
                     Globus RLS
           for File Replica Management

                     Each site represented by a RLS
                        server instance with both LRC
                 Site A            and RLI          Ssite B
                  Site A                                     Ssite B
       rls://sitea.comp.com                       rls://siteb.comp.com
         rls://sitea.comp.com                       rls://siteb.comp.com
fee Þ gsiftp://sitea.comp.com/fee           eef Þ gsiftp://siteb.comp.com/eef
fii Þ gsiftp://sitea.comp.com/fii           iif Þ gsiftp://siteb.comp.com/iif
foo Þ gsiftp://sitea.comp.com/foo           oof Þ gsiftp://siteb.comp.com/oof
fum Þ gsiftp://sitea.comp.com/fum           muf Þ gsiftp://siteb.comp.com/muf
local replica catalog (LRC)                 local replica catalog (LRC)

  rls://siteb.comp.com                          rls://sitea.comp.com
  Þ eef, iif, oof, muf                          Þ fee, fii, foo, fum
replica location index (RLI)                 replica location index (RLI)




                                                                                120
       Finding Files Across the Grid
                                           Which RLSavailable know
                                           File foo is servers at
                                               What do you know
                                         rls://sitea.comp.com knows
                                           Which RLSavailable know
                                            File foo is servers at
                                               What do you know
                                         rls://sitea.comp.com knows
                                                  about file foo?
                                                   about file foo
                                        gsiftp://sitea.comp.com/foo
                                                  about file foo?
                                                              foo
                                        gsiftp://sitea.comp.com/foo




                site A                              site B
                  site A                              site B
      rls://sitea.comp.com                rls://siteb.comp.com
        rls://sitea.comp.com                rls://siteb.comp.com
fee Þ gsiftp://sitea.comp.com/fee   fee Þ gsiftp://siteb.comp.com/eef
fii Þ gsiftp://sitea.comp.com/fii   fii Þ gsiftp://siteb.comp.com/iif
foo Þ gsiftp://sitea.comp.com/foo   foo Þ gsiftp://siteb.comp.com/oof
fum Þ gsiftp://sitea.comp.com/fum   fum Þ gsiftp://siteb.comp.com/muf
local replica catalog (LRC)         local replica catalog (LRC)

       rls://siteb.comp.com                 rls://sitea.comp.com
       Þ eef, iif, oof, muf                 Þ fee, fii, foo, fum

     replica location index (RLI)        replica location index (RLI)




                                                                        121
                Globus RLS Features
l   Soft state update from LRCs to RLIs
    – Relaxed consistency of index
    – Tunable depending on desired load
l   Two alternative update methods supported
    – Full list updates send entire list of LFNs
      periodically, partial updates in between
       > Complete list means always accurate
       > Large lists put drain on network, CPU, storage

    – Optional compressed bloom filter or hash
       > Compression relieves load on network, CPU, storage
       > False positives are possible (tunable rate)

l   Secure access
                                                              122
              RLS Mechanics

l   Command line tools
    – globus-rls-admin: administration and on the
      fly configuration changes
    – globus-rls-cli: simple command line client for
      interacting with both LRC and RLI part of
      server
l   Client APIs
    – C and Java APIs available
    – Functions to publish mappings, query,
      wildcard queries, administration tasks
    – “Bulk” versions of functions for publishing
      and queries on many objects                 123
                 Reliable Wide Area Data
                       Replication
                LIGO Gravitational Wave Observatory



                                          Birmingham•
                                                  §Cardiff




                                             AEI/Golm




Replicating >1 Terabyte/day to 8 sites
>30 million replicas so far
MTBF = 1 month www.globus.org/solutions           124
                The Challenge

Replicate 1 TB/day of data to 10+
international sites.
  – Provide scientists with the means to specify
    and discover data based on application
    criteria (metadata)
  – Provide scientists with the means to locate
    copies of data




                                                   125
          Basic Replication Selection
                 Architecture
                 Attribute                Unique
                 Specification            File Name
 Metadata                                               Replica
 Catalog             1     Application                  Catalog
                                                         File Locations
   Unique File Names
                                         Selected
                                         Replica      Replica               MDS
                                                      Selection
                                                          Performance
                                   File transfer
                                   commands
                                                          Information &
                                                          Predictions
                                                                            NWS


                                 Disk Cache

                             Tape Library
   Disk Array                                             Disk Cache
Replica Location 1          Replica Location 2         Replica Location 3



                                                                            126
          Basic Replication Selection
                 Architecture
                 Attribute               Unique
                 Specification           File Name
 Metadata                                              Replica
 Catalog             1     Application                 Catalog
                                                       File Locations
   Unique File Names
                                        Selected
                                        Replica      Replica               MDS
                                                     Selection
                                                         Performance
                                  File transfer
                                  commands
                                                         Information &
                                                         Predictions
                                                                           NWS

      Metadata Catalog
         maps a set of Disk Cache
     attributes to a set of
                         Tape Library
      unique file names
   Disk Array                                            Disk Cache
Replica Location 1          Replica Location 2        Replica Location 3



                                                                           127
          Basic Replication Selection
                 Architecture
                 Attribute                Unique
                 Specification            File Name
 Metadata                                               Replica
 Catalog                   Application                  Catalog
                                                        File Locations
   Unique File Names
                                         Selected
                                                      Replica               MDS
                       2                 Replica
                                                      Selection
                                                          Performance
                                   File transfer
                                   commands
                                                          Information &
                                                          Predictions
                                                                            NWS


                                 Disk Cache

                             Tape Library
   Disk Array                                             Disk Cache
Replica Location 1          Replica Location 2         Replica Location 3



                                                                            128
          Basic Replication Selection
                 Architecture
                 Attribute                Unique
                 Specification            File Name
 Metadata                                                Replica
 Catalog                   Application
                                                   3     Catalog
                                                         File Locations
   Unique File Names
                                         Selected
                                         Replica       Replica               MDS
                                                       Selection
                                                           Performance
                                   File transfer
                                   commands
                                                           Information &
                                                           Predictions
                                                                             NWS


                                 Disk Cache

                             Tape Library
   Disk Array                                              Disk Cache
Replica Location 1          Replica Location 2          Replica Location 3



                                                                             129
          Basic Replication Selection
                 Architecture
                 Attribute                Unique
                 Specification            File Name
 Metadata                                                Replica
 Catalog                   Application
                                                   3     Catalog
                                                         File Locations
   Unique File Names
                                         Selected
                                         Replica       Replica               MDS
                                                       Selection
                                                           Performance
                                   File transfer
                                   commands
                                                           Information &
                                                           Predictions
                                                                             NWS

                                                     Replica Catalog
                                 Disk Cache         maps a unique file
                             Tape Library
                                                   name to a set of file
   Disk Array
                                                        locations
                                                          Disk Cache
Replica Location 1          Replica Location 2          Replica Location 3



                                                                             130
          Basic Replication Selection
                 Architecture
                 Attribute               Unique
                 Specification           File Name
 Metadata                                              Replica
 Catalog                   Application                 Catalog
                                                       File Locations
   Unique File Names
                                        Selected
                                        Replica      Replica      4        MDS
                                                     Selection
              Replica Selection                          Performance
             uses info sources to
                               File transfer
                               commands
                                                         Information &
                                                         Predictions
                                                                           NWS
             select best file from
                     a set
                                 Disk Cache

                             Tape Library
   Disk Array                                            Disk Cache
Replica Location 1          Replica Location 2        Replica Location 3



                                                                           131
          Basic Replication Selection
                 Architecture
                 Attribute                Unique
                 Specification            File Name
 Metadata                                                Replica
 Catalog                   Application                   Catalog
                                                         File Locations
   Unique File Names
                                         Selected
                                         Replica       Replica               MDS
                                                       Selection
                                                   5       Performance
                                   File transfer
                                   commands
                                                           Information &
                                                           Predictions
                                                                             NWS


                                 Disk Cache

                             Tape Library
   Disk Array                                              Disk Cache
Replica Location 1          Replica Location 2          Replica Location 3



                                                                             132
          Basic Replication Selection
                 Architecture
                 Attribute                  Unique
                 Specification              File Name
 Metadata                                                 Replica
 Catalog                   Application                    Catalog
                                                          File Locations
   Unique File Names
                                           Selected
                                           Replica      Replica               MDS
                                                        Selection
                                                            Performance
                                     File transfer          Information &     NWS
                                 6   commands               Predictions




                                 Disk Cache

                             Tape Library
   Disk Array                                               Disk Cache
Replica Location 1          Replica Location 2           Replica Location 3



                                                                              133
             Basic Replication Selection
                    Architecture
Project Specific                                       General Infrastructure
                    Attribute                Unique
                    Specification            File Name
    Metadata                                               Replica
    Catalog                   Application                  Catalog
                                                            File Locations
      Unique File Names
                                            Selected
                                            Replica      Replica               MDS
                                                         Selection
                                                             Performance
                                      File transfer
                                      commands
                                                             Information &
                                                             Predictions
                                                                               NWS


                                    Disk Cache

                                Tape Library
      Disk Array                                             Disk Cache
   Replica Location 1          Replica Location 2         Replica Location 3



                                                                               134
       LIGO Data Grid: Before & After
Before:                               After:
l   Data replication via              l   24 x 7 x 365 continuous fault
    “FedEx” Grid                          tolerant data streaming
l   Ad-hoc site-by-site               l   Single client tool for scientists
    idioms for finding data               and applications to find data
l   Ad-hoc error prone                l   Scientists concentrate on
    mapping from metadata                 metadata and not file names
    to file names                     l   Multi-site planning of
l   Workflow limited to a                 workflows across LIGO Data
    single resource site                  Grid

    LIGO scientists searching for signals from neutron stars and black holes
    run more jobs across more resources and access more data using the
    LIGO Data Grid.

    Papers are published faster due to the LIGO Data Grid.
                                                                          135
    Another Data Management Use Case

l   Instead of accessing replicated files – what
    if you’re working with a distributed
    database?




                                                   136
OGSA DAI
                      OGSA-DAI
l   Grid Interfaces to Databases
    – Data access
       > Relational & XML Databases, semi-structured files
    – Data integration
       > Multiple data delivery mechanisms, data translation
l   Extensible & Efficient framework
    – Request documents contain multiple tasks
       > A task = execution of an activity
       > Group work to enable efficient operation
    – Extensible set of activities
       > > 30 predefined, framework for writing your own
    – Moves computation to data
    – Pipelined and streaming evaluation
    – Concurrent task evaluation

                                                               138
                  OGSA-DAI

l   Provide service-based access to structured
    data resources as part of Globus
l   Specify a selection of interfaces tailored to
    various styles of data access—starting with
    relational and XML




                                                 139
The OGSA-DAI Framework
 Application
  Application
                  Client Toolkit
                   Client Toolkit




                 OGSA-DAI service

                      Engine

       readFile
        XPath
      SQLQuery         GZip
                       XSLT         GridFTP   Activities


                                                Data
         JDBC         XMLDB          File
                                              Resources


         SQL                        SWISS      Data-
         DB2
        MySQL        XIndice
        Server                      PROT       bases
                                                  140
               OGSA-DAI Features
l   Supports
    – variety of data access, insert and update
    – multiple data delivery mechanisms
    – data transformation
l   Provides support for X509 based security
l   Provides client framework for application developers
l   Highly extensible framework




                                                      141
       OGSA-DAI: Other Features

l   A framework for building data clients
    – Client toolkit library for application
      developers
l   A framework for developing functionality
    – Extend existing activities, or implement
      your own
    – Mix and match activities to provide
      functionality you need
l   Highly extensible
    – Customise the out-of-the-box product
    – Provide your own services, client-side
      support, and data-related functionality
                                                 142
Execution Management
More Specifically, I May Want To …

l   Create a service for use by my colleagues
l   Manage who is allowed to access my
    service (or my experimental data or …)
l   Ensure reliable & secure distribution of
    data from my lab to my partners
l   Run 10,000 jobs on whatever computers I
    can get hold of
l   Monitor the status of the different
    resources to which I have access


                                                144
         Globus Toolkit:
  Open Source Grid Infrastructure
                   Data                      Globus Toolkit v4
                 Replication                  www.globus.org
                                    Grid
 Credential         Replica
                                 Telecontrol
   Mgmt            Location
                                   Protocol
                                 Community
                 Data Access                                Python
 Delegation                      Scheduling      WebMDS
                 & Integration                              Runtime
                                 Framework
                   Reliable
 Community                        Workspace                   C
                     File                        Trigger
Authorization                    Management                 Runtime
                   Transfer
                                 Grid Resource
Authentication                                               Java
                   GridFTP        Allocation &    Index
Authorization
                                  Management                Runtime

                    Data         Execution         Info     Common
 Security
                    Mgmt           Mgmt          Services   Runtime
                                                                      145
            Traditional Resource
           Management Approach

l   Have access to numerous sites
    – Accounts, permissions, etc
l   Use a Metascheduler to make resource
    selection decisions
    – GridWay
    – Metascheduler uses GRAM to contact the
      difference local queuing systems




                                               146
    Execution Management (GRAM)

l   Common WS interface to schedulers
    – Unix, Condor, LSF, PBS, SGE, …
l   More generally: interface for process
    execution management
    – Lay down execution environment
    – Stage data
    – Monitor & manage lifecycle
    – Kill it, clean up




                                            147
          GRAM - Basic Job
    Submission and Control Service
l   A uniform service interface
    for remote job submission
    and control
    – Includes file staging and I/O
      management
    – Includes reliability features
    – Supports basic Grid security
      mechanisms
    – Available in Pre-WS and WS
l   GRAM is not a scheduler.
    – No scheduling
    – No metascheduling/brokering
    – Often used as a front-end to
      schedulers, and often used to
      simplify metaschedulers/brokers   148
         GRAM4 (aka WS GRAM)

l   2nd-generation WS implementation
    optimized for performance, flexibility,
    stability, scalability
l   Streamlined critical path
    – Use only what you need
l   Flexible credential management
    – Credential cache & delegation service
l   GridFTP & RFT used for data operations
    – Data staging & streaming output
    – Eliminates redundant GASS code

                                              149
    Using GRAM vs Building a Service

l   GRAM is intended for jobs that
     – are arbitrary programs
     – need stateful monitoring or credential
       management
     – Where file staging is important
l   If the application is lightweight, with
    modest input/output, may be a better
    candidate for hosting directly as a WSRF
    service



                                                150
                           GRAM4 Architecture
                                  Service host(s) and compute element(s)

                                              Job events
                                                                SEG
                     Globus Java Container                                Compute element
                              GRAM
                             GRAM              Local job control
                             services                                           Local
                            services
         Job tions                                             GRAM




                                                        sudo
                                             Deleg                            scheduler
          func                                    ate
Client




                                  Transfer                     adapter
                     Delegation   request
         Delegate
                                                               GridFTP        User
                            RFT File
                                              FTP                              job
                            Transfer
                                              control
                                                                   FTP data
                                                                              Remote
                                                               GridFTP        storage
                                                                              element(s)

                                                                                     151
                           GRAM4 Architecture
                                  Service host(s) and compute element(s)

                                              Job events
                                                                SEG
                     Globus Java Container                                Compute element
                              GRAM
                             GRAM              Local job control
                             services                                           Local
                            services
         Job tions                                             GRAM




                                                        sudo
                                             Deleg                            scheduler
          func                                    ate
Client




                                  Transfer                     adapter
                     Delegation   request
         Delegate
                                                               GridFTP        User
                            RFT File
                                              FTP                              job
                            Transfer
                                              control
                                                                   FTP data
                                                                              Remote
                                                               GridFTP        storage
    Delegated credential can be:                                              element(s)
    Made available to the application
                                                                                     152
                           GRAM4 Architecture
                                  Service host(s) and compute element(s)

                                              Job events
                                                                SEG
                     Globus Java Container                                Compute element
                              GRAM
                             GRAM              Local job control
                             services                                           Local
                            services
         Job tions                                             GRAM




                                                        sudo
                                             Deleg                            scheduler
          func                                    ate
Client




                                  Transfer                     adapter
                     Delegation   request
         Delegate
                                                               GridFTP        User
                            RFT File
                                              FTP                              job
                            Transfer
                                              control
                                                                   FTP data
                                                                              Remote
                                                               GridFTP        storage
    Delegated credential can be:                                              element(s)
    Used to authenticate with RFT
                                                                                     153
                           GRAM4 Architecture
                                  Service host(s) and compute element(s)

                                              Job events
                                                                SEG
                     Globus Java Container                                Compute element
                              GRAM
                             GRAM              Local job control
                             services                                           Local
                            services
         Job tions                                             GRAM




                                                        sudo
                                             Deleg                            scheduler
          func                                    ate
Client




                                  Transfer                     adapter
                     Delegation   request
         Delegate
                                                               GridFTP        User
                            RFT File
                                              FTP                              job
                            Transfer
                                              control
                                                                   FTP data
                                                                              Remote
                                                               GridFTP        storage
    Delegated credential can be:                                              element(s)
    Used to authenticate with GridFTP
                                                                                     154
                  GRAM4 features
l   At most one submission
l   Staging:
    – In data and out results
    – Uses RFT and hence GridFTP
l   Credentials
    – Delegation for job submit
    – Staging data credentials
l   Batch submission
    – Poll for status using endpoint
    – Subscribe to notifications
l   Specify scheduler options
l   Mutli job submissions


                                       155
          GRAM Tool Mechanics

l   globusrun-ws
l   -s submit
l   -c command line to run
l   -S streaming of data




                                156
    Resource Specification Language (RSL)

l    For more complicated jobs, we’ll use RSL to specify
     the job
<job>
<executable>/bin/echo</executable>
<argument>this is an example_string </argument>
<argument>Globus was here</argument>
<stdout>${GLOBUS_USER_HOME}/stdout</stdout>
    <stderr>${GLOBUS_USER_HOME}/stderr</stderr>
</job>




                                                       157
             RSL Substitutions
l   GRAM will perform some variable
    substitutions for you
    –   GLOBUS_USER_HOME
    –   GLOBUS_USER_NAME
    –   GLOBUS_SCRATCH_DIR
    –   GLOBUS_LOCATION
l   SCRATCH_DIR will be a compute-node
    local high-speed storage if defined, or
    GLOBUS_USER_HOME if not



                                              158
                 GridWay Meta-Scheduler
      l   Scheduler virtualization layer on top of Globus
          services
          – A LRM-like environment for submitting, monitoring,
            and controlling jobs
          – A way to submit jobs to the Grid, without having to
            worry about the details of exactly which local
            resource will run the job
          – A policy-driven job scheduler, implementing a
            variety of access and Grid-aware load balancing
            policies
          – Accounting




GridWay: http://www.gridway.org                                   159
                   DRMAA library                 CLI
                                                             Job Submission
                                                              Job Submission
                                                             Job Monitoring
                                                              Job Monitoring
                        GridWay Core          Request        Job Control
                                                              Job Control
                                              Manager        Job Migration
                                                              Job Migration
                    Job Pool     Host Pool
                                              Dispatch
                                                                    Scheduler
                                              Manager


                        Transfer             Execution         Information
                        Manager               Manager            Manager

                                             pre-WS    WS            MDS2
                    GridFTP    RFT                           MDS2           MDS4
                                             GRAM     GRAM           GLUE
      Job Preparation
      Job Preparation
      Job Termination                                                         Resource Discovery
                                                                              Resource Discovery
      Job Termination
      Job Migration                                                           Resource Monitoring
                                                                              Resource Monitoring
      Job Migration
                         Grid                  Grid                Grid
                    File Transfer            Execution         Information
                      Services               Services            Services
GridWay: http://www.gridway.org                                                            160
Monitoring and Discovery
More Specifically, I May Want To …

l   Create a service for use by my colleagues
l   Manage who is allowed to access my
    service (or my experimental data or …)
l   Ensure reliable & secure distribution of
    data from my lab to my partners
l   Run 10,000 jobs on whatever computers I
    can get hold of
l   Monitor the status of the different
    resources to which I have access


                                                162
         Globus Toolkit:
  Open Source Grid Infrastructure
                   Data                      Globus Toolkit v4
                 Replication                  www.globus.org
                                    Grid
 Credential         Replica
                                 Telecontrol
   Mgmt            Location
                                   Protocol
                                 Community
                 Data Access                                Python
 Delegation                      Scheduling      WebMDS
                 & Integration                              Runtime
                                 Framework
                   Reliable
 Community                        Workspace                   C
                     File                        Trigger
Authorization                    Management                 Runtime
                   Transfer
                                 Grid Resource
Authentication                                               Java
                   GridFTP        Allocation &    Index
Authorization
                                  Management                Runtime

                    Data         Execution         Info     Common
 Security
                    Mgmt           Mgmt          Services   Runtime
                                                                      163
      Monitoring and Discovery System
                  (MDS4)

l   Grid-level monitoring system
    – Aid user/agent to identify host(s) on which
      to run an application
    – Warn on errors
l   Uses standard interfaces to provide
    publishing of data, discovery, and data
    access, including subscription/notification
    – WS-ResourceProperties, WS-
      BaseNotification, WS-ServiceGroup
l   Functions as an hourglass to provide a
    common interface to lower-level
    monitoring tools                                164
                 Information Users :
      Schedulers, Portals, Warning Systems, etc.




                                          WS standard
                                          interfaces for
Standard Schemas                          subscription,
(GLUE schema, eg)                         registration,
                                          notification


Cluster monitors
(Ganglia, Hawkeye,
Clumon, and                              Queuing systems
Nagios)                  Services       (PBS, LSF, Torque)
                     (GRAM, RFT, RLS)                165
              MDS4 Components
l   Information providers
    – Monitoring is a part of every WSRF service
    – Non-WS services are also be used
l   Higher level services
    – Index Service – a way to aggregate data
    – Trigger Service – a way to be notified of changes
    – Both built on common aggregator framework
l   Clients
    – WebMDS


l   All of the tool are schema-agnostic, but
    interoperability needs a well-understood
    common language
                                                          166
                            GT4
                   Monitoring & Discovery
                                           Clients (e.g., WebMDS)
                     GT4 Container
WS-ServiceGroup
                         MDS-
                         Index
 Registration &
WSRF/WSN Access
                          adapter

                                                  GT4 Cont.
   GT4 Container
          MDS-              Custom protocols
                                                   MDS-
          Index           for non-WSRF entities    Index
    Automated
   registration                      GridFTP
   in container
                                                     RFT
   GRAM           User
                                                            167
          Information Providers
l   Data sources for the higher-level services
l   Some are built into services
    – Any WSRF-compliant service publishes
      some data automatically
    – WS-RF gives us standard
      Query/Subscribe/Notify interfaces
    – Globus services: ServiceMetaDataInfo
      element includes start time, version, and
      service type name
    – Most of them also publish additional useful
      information as resource properties



                                                    168
          Information Providers:
             Globus Services
l   Reliable File Transfer Service (RFT)
    – Service status data, number of active
      transfers, transfer status, information about
      the resource running the service
l   Community Authorization Service (CAS)
    – Identifies the VO served by the service
      instance
l   Replica Location Service (RLS)
    – Note: not a WS
    – Location of replicas on physical storage
      systems (based on user registrations) for
      later queries
                                                  169
         Information Providers

l   Other sources of data
    – Any executables
    – Other (non-WS) services
    – Interface to another archive or data
      store
    – File scraping
l   Just need to produce a valid XML
    document


                                             170
           Information Providers:
           Cluster and Queue Data
l   Interfaces to Hawkeye, Ganglia, CluMon, Nagios
    – Basic host data (name, ID), processor information,
      memory size, OS name and version, file system
      data, processor load data
    – Some condor/cluster specific data
    – This can also be done for sub-clusters, not just at
      the host level
l   Interfaces to PBS, Torque, LSF
    – Queue information, number of CPUs available and
      free, job count information, some memory statistics
      and host info for head node of cluster




                                                            171
          Higher-Level Services
l   Index Service
    – Caching registry
l   Trigger Service
    – Warn on error conditions
l   Archive Service
    – Database store for history (in development)


l   All of these have common needs, and are
    built on a common framework



                                                    172
      Common Aggregator Framework

l   Basic framework for higher-level
    functions
    – Subscribe to Information Provider(s)
    – Do some action
    – Present standard interfaces




                                             173
 Aggregator Framework Features

1) Common configuration mechanism
  – Specify what data to get, and from where
2) Self cleaning
  – Services have lifetimes that must be
    refreshed
3) Soft consistency model
  – Published information is recent, but not
    guaranteed to be the absolute latest
4) Schema Neutral
  – Valid XML document needed only
                                               174
            MDS4 Index Service
l   Index Service is both registry and cache
    – Datatype and data provider info, like a
      registry (UDDI)
    – Last value of data, like a cache
l   In memory default approach
    – DB backing store currently being developed
      to allow for very large indexes
l   Can be set up for a site or set of sites, a
    specific set of project data, or for user-
    specific data only
l   Can be a multi-rooted hierarchy
    – No *global* index

                                                   175
           MDS4 Trigger Service

l   Subscribe to a set of resource properties
l   Evaluate that data against a set of pre-
    configured conditions (triggers)
l   When a condition matches, action occurs
    – Email is sent to pre-defined address
    – Website updated


l   Similar functionality in Hawkeye



                                                176
         WebMDS User Interface
l   Web-based interface to WSRF resource
    property information
l   User-friendly front-end to Index Service
l   Uses standard resource property requests
    to query resource property data
l   XSLT transforms to format and display
    them
l   Customized pages are simply done by
    using HTML form options and creating your
    own XSLT transforms
l   Sample page:
    – http://mds.globus.org:8080/webmds/webm
      ds?info=indexinfo&xsl=servicegroupxsl
                                            177
178
          Working with TeraGrid
l   Large US project across 9 different sites
    – Different hardware, queuing systems and
      lower level monitoring packages
l   Starting to explore MetaScheduling
    approaches
l   Need a common source of data with a
    standard interface for basic scheduling info




                                                   179
                Data Collected

l   Provide data at the subcluster level
    – Sys admin defines a subcluster, we query
      one node of it to dynamically retrieve
      relevant data
l   Can also list per-host details
l   Interfaces to Ganglia, Hawkeye, CluMon,
    and Nagios available now
    – Other cluster monitoring systems can write
      into a .html file that we then scrape
l   Also collect basic queuing data, some
    TeraGrid specific attributes
                                                   180
181
                    DOE Earth System Grid

Goal: Enable
sharing &
analysis of
high-volume
data from
advanced
earth system
models



www.earthsystemgrid.org                     182
                          ESG Technologies
                                                                NCAR                                 ORNL
     Climate data
                                                      NCAR
 l                                                    Cache     MSS                                  HPSS


      – Metadata catalog                               RLS       SRM                             RLS        SRM


      – OPenDAP-G (aggregation and
                                                              MyProxy        SRM    DISK    OPeNDAP-G
        subsetting)                                                                 Cache

                                              SRM
 l   Data management                NERSC
                                                                          ESG Web Portal
                                                                                                                    RLS   LANL
                                                                                                                          Cache
                                               RLS               User          Catalogs          Data
      – Data Mover Lite                                       Registration     Browsing         Search
                                                                Access         Climate           Data
      – Storage Resource Manager                                Control        Metadata        Download
                                                                Data             Data           Usage
      – Globus Security Infrastructure                        Subsetting       Publishing       Metrics
                                                                              Monitoring
                                                                               Services
      – GridFTP
      – Globus Replica Location Service                                              search
                                                      Web
                                                      Web                                             Web
                                                                                                      Web
                                                                  publish            browse         Browser
 l   Security services                     Data
                                                    Browser
                                                    Browser                         download
                                                                                                    Browser
                                                                                                                  Data
                                                                                                       DML
                                                                                                       DML        User
                                         Provider
      – Access control
      – MyProxy                                      MSS, HPSS: Tertiary
      – PURSE User registration                      data storage systems
Dave Bernholdt, ORNL                                                                                                      183
        Monitoring Overall System Status

  l   Monitored data are collected in
      MDS4 Index service
  l   Information providers
      check resource status at a
      configured frequency
      – Currently, every 10 min
  l   Report status to Index
  l   Information in Index is queried by
      ESG Web portal
  l   Used to generate overall picture of
      state of ESG
  l   Displayed on ESG Web portal


Ann Chervenak, USC/ISI                      184
          ESG: Warning on Errors Sample

      Total error messages for May 2006                   47
      Messages related to certificate and configuration   38
        problems at LANL
      Failure messages due to brief interruption in        2
         network service at ORNL on 5/13
      HTTP data server failure at NCAR 5/17                1
      RLS failure at LLNL 5/22                             1
      Simultaneous error messages for SRM services at      3
        NCAR, ORNL, LBNL on 5/23
      RLS failure at ORNL 5/24                             1
      RLS failure at LBNL 5/31                             1


Ann Chervenak, USC/ISI                                     185
GT4 Distribution
               Globus Software: dev.globus.org
     Globus Projects
                                                         OGSA-DAI      GT4
     MPICH-
       G2               Java                               Data      Replica
                                 Delegation   MyProxy
                       Runtime                             Rep      Location

     GridWay             C                      GSI-
                                    CAS                  GridFTP       MDS4
                       Runtime                OpenSSH

    Incubator                                            Reliable
      Mgmt             Python
                                   C Sec       GRAM        File     GT4 Docs
                       Runtime
                                                         Transfer


 Incubator
 Projects              GAARDS MEDICUS Cog WF Virt WkSp
 GDTE       GridShib    OGRO     UGP       Dyn Acct Gavia JSC   DDM     Metrics
Introduce   PURSE      HOC-SA    LRMA       WEEP    Gavia MS    SGGC   ServMark

Common                     Execution                       Info
Runtime
               Security
                             Mgmt
                                           Data Mgmt
                                                         Services          187
                                                                        Other
              Our Goals for GT4

l   Usability, reliability, scalability, …
    – Web service components have quality equal
      or superior to pre-WS components
    – Documentation at acceptable quality level
l   Consistency with latest standards (WS-*,
    WSRF, WS-N, etc.) and Apache platform
    – WS-I Basic Profile compliant
    – WS-I Basic Security Profile compliant
l   New components, platforms, languages
    – And links to larger Globus ecosystem

                                                  188
                  Why Use GT4?
l   Performance and reliability
    – Literally millions of tests and queries run against GT4
      services
l   Scalability
    – Many lessons learned from GT2 have been addressed
      in GT4
l   Support
    – This is our active code base, much more attention
l   Additional functionality
    – New features are here
    – Additional GRAM interfaces to schedulers, MDS
      Trigger service, GridFTP protocol interfaces, etc
l   Easier to contribute to
                                                            189
        Versioning and Support

l   Versioning
    – Evens are production (4.0.x, 4.2.x),
    – Odds are development (4.1.x)
l   We support this version and the one
    previous
    – Currently stable version 4.0.5
    – We support 3.2.x and 4.0.x
    – We’ve also got the 4.1.3 dev release
      available
                                             190
         Several “Next” Versions
l   4.0.6 – stable release
    – 100% same interfaces
    – Expected early Q1 2008
l   4.1.4 – development release(s)
    – New functionality
    – Expected every 6-8 weeks
l   4.2.0 - stable release
    – Tested, documented 4.1.x branch
    – Likely late summer or early fall
    – Discussed on gt-dev@globus.org
l   5.0 – substantial code base change
    – With any luck, not for years :)
                                         191
             Tested Platforms
l   Debian                l   SGI Altix (IA64
l   Fedora Core               running Red Hat)
l   FreeBSD               l   SuSE Linux
l   HP/UX                 l   Tru64 Unix
l   IBM AIX               l   Apple MacOS X (no
l   Red Hat                   binaries)
l   Sun Solaris           l   Windows – Java
                              components only

List of binaries and known platform-specific
   install bugs at
http://www.globus.org/toolkit/docs/4.0/admin/
  docbook/ ch03.html
                                                  192
             Documentation Overview
l       Current document significantly more detailed
        than earlier versions
        – http://www.globus.org/toolkit/docs/4.0/
l       Tutorials available for those of you building a
        new service
        – http://www-unix.globus.org/toolkit/tutorials/BAS/


    l   Globus® Toolkit 4: Programming
        Java Services (The Morgan
        Kaufmann Series in Networking),
        by Borja Sotomayor, Lisa Childers
        (Available through Amazon, £19.99
        or $20)
                                                              193
    General Globus Help and Support

l   Globus toolkit help lists list
    – gt-user@globus.org
    – gt-dev@globus.org
    – http://dev.globus.org/wiki/
       Mailing_Lists
l   Each project has specific lists
l   Bugzilla
    – bugzilla.globus.org


                                      194
dev.globus.org
               Globus Software: dev.globus.org
     Globus Projects
                                                         OGSA-DAI      GT4
     MPICH-
       G2               Java                               Data      Replica
                                 Delegation   MyProxy
                       Runtime                             Rep      Location

     GridWay             C                      GSI-
                                    CAS                  GridFTP       MDS4
                       Runtime                OpenSSH

    Incubator                                            Reliable
      Mgmt             Python
                                   C Sec       GRAM        File     GT4 Docs
                       Runtime
                                                         Transfer


 Incubator
 Projects              GAARDS MEDICUS Cog WF Virt WkSp
 GDTE       GridShib    OGRO     UGP       Dyn Acct Gavia JSC   DDM     Metrics
Introduce   PURSE      HOC-SA    LRMA       WEEP    Gavia MS    SGGC   ServMark

Common                     Execution                       Info
Runtime
               Security
                             Mgmt
                                           Data Mgmt
                                                         Services          196
                                                                        Other
    Incubator Process in dev.globus

l   Entry point for new Globus projects
l   Incubator Management Project (IMP)
    – Oversees incubator process form first
      contact to becoming a Globus project
    – Quarterly reviews of current projects
    – Process being debugged by “Incubator
      Pioneers”
http://dev.globus.org/wiki/Incubator/
  Incubator_Process



                                              197
       Incubator Process (1 of 3)
l   Project proposes itself as a Candidate
    – A proposed name for the project;
    – A proposed project chair, with contact info;
    – A list of the proposed committers for the
      project;
    – An overview of the aims of the project;
    – An overview of any current user base or
      user community, if applicable;
    – An overview of how the project relates to
      other parts of Globus;
    – A summary of why the project would
      enhance and benefit Globus.


                                                     198
       Incubator Process (2 of 3)
l   IMP meet, discuss, and accept project
    as a Incubator Project
    – Project is now part of the Incubator
      framework
    – Get assigned a Mentor to help
       >Member of IMP
       >Bridge between Globus and new
         Incubator Project
    – Opportunity to get up to speed on Globus
      Development process



                                                 199
       Incubator Process (3 of 3)

l   Quarterly reviews by IMP determine
    – Stay an Incubator Project
    – Retire
    – Escalate to a full Globus project
l   Escalation when Project passes checklist
    – Legal
    – Meritocracy
    – Alignment/Synergy
    – Infrastructure


                                           200
       Current Incubator Projects
            dev.globus.org

l   Distributed Data     l   GridShib            l   Portal-based
    Management (DDM) l       Grid Toolkit Handle     User Registration
                             System (gt-hs)          Service (PURSe)
l   Dynamic Accounts
                         l   Higher Order        l   ServMark
l   Gavia-Meta
                             Component Service l     SJTU GridFTP
    Scheduler                Architecture (HOC-      GUI Client
l   Gavia- Job               SA)                     (SGGC)
    Submission Client l      Introduce           l   UCLA Grid Portal
l   Grid Authentication l    Local Resource          Software (UGP)
    and Authorization        Manager Adaptors l      WEEP
    with Reliably            (LRMA)              l   Cog Workflow
    Distributed Services l   Metrics             l   Virtual
    (GAARDS)             l   MEDICUS                 Workspaces
l   Grid Development l       Open GRid OCSP
    Tools for Eclipse        (Online Certificate
    (GDTE)                   Status Protocol)

                                                                   201
                  Security Committee
   l   Goals
       – Evaluate and resolve security vulnerabilities prior to
         making it public
       – Potential vulnerabilities: sec-alert@globus.org
   l   Membership
       – Any dev.globus committer
       – Subscribed to sec-committee@globus.org
       – Owns vulnerabilities and has voting rights
   l   Lurkers
       – Participate in discussions




                                                                          202
http://dev.globus.org/wiki/SecurityCommittee/Security_Vulnerability_Handling
           Security Committee

l   Membership requires approval
    – Majority quorum amongst members
l   Participating communities
    – Receive advance notice of advisory
    – TeraGrid, VDT, Condor
l   Community inclusion request
    – Nominated and voted on by members
    – GT usage and participation in committee
      activities


                                                203
Summary
       Summary: Grids are About …


Enabling “coordinated resource sharing
& problem solving in dynamic, multi-
institutional virtual organizations.”
     (Source: “The Anatomy of the Grid”)


 l   Access to shared resources
      à Virtualization, allocation, management
 l   With predictable behaviors
      à Provisioning, quality of service
 l   In dynamic, heterogeneous environments
      à Standards-based interfaces and protocols

                                                   205
              … By Providing
            Open Infrastructure

l   Web services standards
    – State, notification, security, …
l   Services that enable access to resources
    – Service-enable new & existing resources
    – E.g., GRAM on computer, GridFTP on
      storage system, custom application services
    – Uniform abstractions & mechanisms
l   Tools to build applications that exploit this
    infrastructure
    – Registries, security, data management, …
l   A rich tool & service ecosystem                 206
            More Specifically,
          Making it Possible to …

l   Create a service for use by my colleagues
l   Manage who is allowed to access my
    service (or my experimental data or …)
l   Ensure reliable & secure distribution of
    data from my lab to my partners
l   Run 10,000 jobs on whatever computers I
    can get hold of
l   Monitor the status of the different
    resources to which I have access
l   And so on …

                                                207
    Contribute to an Existing Project

l   Contribute code, documentation, design
    ideas, and feature requests
l   Joining the mailing lists
    – *-dev, *-user, *-announce for each project
    – See the project wiki page at dev.globus.org
l   Chime in at any time
l   Regular contributors can become
    committers, with a role in defining project
    directions
http://dev.globus.org/wiki/How_to_contribute

                                                    208
             For More Information

l   Rachana Ananthakrishnan
    – ranantha@mcs.anl.gov
l   Globus Alliance
    – http://www.globus.org
l   Dev.globus
    – http://dev.globus.org
l   Upcoming Events
    – http://dev.globus.org/wiki/Outreach
l   Globus Solutions
    – http://www.globus.org/solutions/
                                            209

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:0
posted:10/5/2013
language:English
pages:209