Document Sample
Globus Powered By Docstoc
					Globus: A Core Grid Middleware

    Source: The Globus Project
   Argonne National Laboratory,
      University of Southern
          California / ISI

   Localised by: Rajkumar Buyya
           The Globus Project

n   Basic research in grid-related technologies
    n   Resource & data management, security, QoS,
        policy, communication, adaptation, etc.
n   Development of Globus Toolkit
    n   Core services for grid-enabled tools & apps
n   Construction of production grids & testbeds
    n   Multiple deployments to distributed organizations
        for production & prototyping
n   Application experiments
    n   Distributed applications, tele-immersion, etc.
             Globus Approach

n   A toolkit and collection of services addressing
    key technical problems
    n   Modular “bag of services” model
    n   Not a vertically integrated solution
    n   General infrastructure tools (aka middleware) that
        can be applied to many application domains
n   Inter-domain issues, rather than clustering
    n   Integration of intra-domain solutions
n   Distinguish between local and global services
                 Globus Hourglass

n   Focus on architecture issues          Applications

    n   Propose set of core services as Diverse global services
        basic infrastructure
    n   Use to construct high-level,
        domain-specific solutions
n   Design principles                 Core Globus
    n   Keep participation cost low
    n   Enable local control
    n   Support for adaptation
    n   “IP hourglass” model
                                               Local OS
Technical Focus & Approach

n   Enable incremental development of grid-
    enabled tools and applications
    n   Model neutral: Support many programming models,
        languages, tools, and applications
    n   Evolve in response to user requirements
n   Deploy toolkit on international-scale
    production grids and testbeds
    n   Large-scale application development & testing
n   Information-rich environment
    n   Basis for configuration and adaptation
        Globus Toolkit Services

n   Security (GSI)
    n   PKI-based Security (Authentication) Service
n   Job submission and management (GRAM)
    n   Uniform Job Submission
n   Information services (MDS)
    n   LDAP-based Information Service
n   Remote file management (GASS)
    n   Remote Storage Access Service
n   Remote Data Catalogue and Management Tools
    n   Support by Globus 2.0 released in 2002
    Grid Services Architecture
                High-energy          Collaborative            On-line
                physics data          engineering        instrumentation
Applications      analysis     Regional         Parameter
                           climate studies       studies

                   High                     Collab.                 Remote
 Application    throughput                  design                   control
Toolkit Layer                   Data-                  Remote
                              intensive                  viz

Grid Services   Information       Resource mgmt        ...
                  Security        Data access     Fault detection

 Grid Fabric     Transport                ...         Multicast
                Instrumentation       Control interfaces        QoS mechanisms
         Layered Architecture


 GlobusView                 Application Toolkits                        Testbed Status

 DUROC              MPI         Condor-G           HPC++     Nimrod/G         globusrun

                                  Grid Services
         Nexus                                                          GRAM

   I/O              MDS            GSI             GSI-FTP    HBM               GASS

Condor        MPI                   Grid Fabric                         TCP         UDP

 LSF          PBS         NQE              Linux        NT    Solaris           DiffServ
         Sample of High-Level
n   Resource brokers and co-allocators
    n   DUROC, Nimrod/G, Condor-G, GridbusBroker
        Communication & I/O libraries
n   Parallel languages
    n   HPC++, CC++, Nimrod Parameter Specification
n   Collaborative environments
    n   CAVERNsoft, ManyWorlds
n   Others
    n   MetaNEOS, NetSolve, LSA, AutoPilot, WebFlow
The Nimrod-G Grid Resource Broker
 n   A resource broker for managing, steering, and executing
     task farming (parameter sweep/SPMD model)
     applications on the Grid based on deadline and
     computational economy.
 n   Based on users’ QoS requirements, our Broker
     dynamically leases services at runtime depending on
     their quality, cost, and availability.
 n    Key Features
     n   A single window to manage & control experiment
     n   Persistent and Programmable Task Farming Engine
     n   Resource Discovery
     n   Resource Trading
     n   Scheduling & Predications
     n   Generic Dispatcher & Grid Agents
     n   Transportation of data & results
     n   Steering & data management
     n   Accounting
 n   Uses Globus – MDS, GRAM, GSI, GASS
Condor-G: Condor for the Grid
n   Condor is a high-throughput scheduler
n   Condor-G uses Globus Toolkit libraries for:
    n   Security (GSI)
    n   Managing remote jobs on Grid (GRAM)
    n   File staging & remote I/O (GSI-FTP)
n   Grid job management interface & scheduling
    n   Robust replacement for Globus Toolkit programs
         n   Globus Toolkit focus is on libraries and services, not end
             user vertical solutions
    n   Supports single or high-throughput apps on Grid
         n   Personal job manager which can exploit Grid resources
Production Grids & Testbeds

n   Production deployments underway at:
    n   NSF PACIs National Technology Grid
    n   NASA Information Power Grid
    n   DOE ASCI
    n   European Grid
n   Research testbeds
    n   EMERGE: Advance reservation & QoS
    n   GUSTO: Globus Ubiquitous Supercomputing
        Testbed Organization
    n   Particle Physics Data Grid
    n   World-Wide Grid (WWG)
   Production Grids & Testbeds

NASA’s Information Power Grid     The Alliance National Technology Grid

                        GUSTO Testbed
                 World Wide Grid (WWG)
                                                                               North America
                      Melbourne+Monash U:
   GMonitor                                                                   ANL: SGI/Sun/SP2
                                         Gridbus+                             NCSA: Cluster
                                         Nimrod-G                             Wisc: PC/cluster
                                                                              NRC, Canada
                      VPAC, Physics                                           Many others
                                Solaris WS

@ SC 2002/Baltimore
                                                                  WW Grid
                        Grid Market                               Internet   ZIB: T3E/Onyx
                         Directory                                           AEI: Onyx
                                                                             CNR: Cluster
                                                                             CUNI/CZ: Onyx
                                       Asia                                  Pozman: SGI/SP2
                                                                             Vrije U: Cluster
                                      AIST, Japan: Solaris                   Cardiff: Sun E6500
                                                   Cluster                   Portsmouth: Linux PC
                                      Osaka University: Cluster              Manchester: O3K
                                      Doshia: Linux cluster                  Cambridge: SGI
                                      Korea: Linux cluster                   Many others
Example Applications Projects
  (via Nimrod-G or Gridbus)
n   Molecular Docking for Drug Discovery
    n   Docking molecules from chemical databases with
        target protein
n   Neuro Science
    n   Brain Activity Analysis
n   High Energy Physics
    n   Belle Detector Data Analysis
n   Natural Language Engineering
    n   Analyzing audio data (e.g., to identify emotional
        state of a person!)
Example Application Projects

n   Computed microtomography (ANL, ISI)
    n   Real-time, collaborative analysis of data from X-Ray
        source (and electron microscope)
n   Hydrology (ISI, UMD, UT; also NCSA, Wisc.)
    n   Interactive modeling and data analysis
n   Collaborative engineering (“tele-immersion”)
    n   CAVERNsoft @ EVL
    n   Large CFD simulations for aerospace vehicles
           Example Application
n   Distributed interactive simulation (CIT, ISI)
    n   Record-setting SF-Express simulation
n   Cactus
    n   Astrophysics simulation, viz, and steering
    n   Including trans-Atlantic experiments
n   Particle Physics Data Grid
    n   High Energy Physics distributed data analysis
n   Earth Systems Grid
    n   Climate modeling data management
        The Globus Advantage

n   Flexible Resource Specification Language which
    provides the necessary power to express the
    required constraints
n   Services for resource co-allocation, executable
    staging, remote data access and I/O streaming
n   Integration of these services into high-level tools
    n   MPICH-G: grid-enabled MPI
    n   globus-job-*: flexible remote execution commands
    n   Nimrod-G Grid Resource broker
    n   Gridbus: Grid Business Infrastructure
    n   Condor-G: high-throughput broker
    n   PBS, GRD: meta-schedulers
     Resource Management

n   Resource Specification Language (RSL)
    is used to communicate requirements
n   The Globus Resource Allocation
    Manager (GRAM) API allows programs
    to be started on remote resources,
    despite local heterogeneity
n   A layered architecture allows
    application-specific resource brokers
    and co-allocators to be defined in
    terms of GRAM services
      Management Architecture

                       RSL                                    specialization

                                                    Queries     Information
                                                    & Info        Service
                       Ground RSL


                                    Simple ground RSL
Local        GRAM                       GRAM                       GRAM
managers         LSF                   EASY-LL                      NQE
                     GRAM Components
                             MDS client API calls
                             to locate resources
           Client                                    MDS: Grid Index Info Server
                             MDS client API calls                                   Site boundary
                             to get resource info

 GRAM client API calls to
request resource allocation
                                               MDS:   Grid Resource Info Server
   and process creation.                                           Query current status
                          GRAM client API state                    of resource
        Globus Security     change callbacks
                                                    Local Resource Manager
                                                                                Allocate &
                                                                             create processes
                         Create      Job Manager

        Gatekeeper                  Parse
                                                       Monitor &
                                                        control       Process
                                      RSL Library
            A simple run

n   [raj@belle raj]$ globus-job-run /bin/date
n   Mon May 3 15:05:42 EST 2004
Resource Specification Language
 n   Common notation for exchange of
     information between components
     n   Syntax similar to MDS/LDAP filters
 n   RSL provides two types of information:
     n   Resource requirements: Machine type,
         number of nodes, memory, etc.
     n   Job configuration: Directory, executable,
         args, environment
 n   API provided for manipulating RSL
                    RSL Syntax

n   Elementary form: parenthesis clauses
    n   (attribute op value [ value … ] )
n   Operators Supported:
    n   <, <=, =, >=, > , !=
n   Some supported attributes:
    n   executable, arguments, environment, stdin,
        stdout, stderr, resourceManagerContact,
n   Unknown attributes are passed through
    n   May be handled by subsequent tools
           Constraints: “&”

n    globusrun -o -r
n   For example:
    & (count>=5) (count<=10)
      (max_time=240) (memory>=64)

“Create 5-10 instances of myprog, each on a
  machine with at least 64 MB memory that is
  available to me for 4 hours”
          Disjunction: “|”

n   For example:
n      & (executable=myprog)
n        ( | (&(count=5)(memory>=64))
n            (&(count=10)(memory>=32)))
n   Create 5 instances of myprog on a
    machine that has at least 64MB of
    memory, or 10 instances on a machine
    with at least 32MB of memory
            Multirequest: “+”

n   A multi-request allows us to specify multiple
    resource needs, for example
    + (& (count=5)(memory>=64)
      (&(network=atm) (executable=p2))
    n   Execute 5 instances of p1 on a machine with at
        least 64M of memory
    n   Execute p2 on a machine with an ATM connection
n   Multirequests are central to co-allocation

n   Simultaneous allocation of a resource set
    n   Handled via optimistic co-allocation based on free
        nodes or queue prediction
    n   In the future, advance reservations will also be
n   globusrun and globus-job-* will co-allocate
    specific multi-requests
    n   Uses a Globus component called the Dynamically
        Updated Request Online Co-allocator (DUROC)
           DUROC Functions

n   Submit a multi-request
n   Edit a pending request
    n   Add new nodes, edit out failed nodes
n   Commit to configuration
    n   Delay to last possible minute
    n   Barrier synchronization
n   Initialize computation
    n   Bootstrap library
n   Monitor and control collection
        DUROC Architecture
                      RM1      RM2
Subjob status                              Jobs
                       Job 1    Job 2

Controlling     RSL
Application multi-request       Job 3

                                 Job 4

                                 Job 5
       Edit request                      Barrier
RSL Creation Using globus-job-
 n   globus-job-run can be used to generate
     RSL from command-line args:
     globus-job-run –dumprsl \
              -: host1 -np N1 [-s] executable1 args1 \
              -: host2 -np N2 [-s] executable2 args2 \
                  ... > rslfile
     n   -np: number of processors
     n   -s: stage file
     n   argument options for all RSL keywords
     n   -help: description of all options
Job Submission Interfaces

n   Globus Toolkit includes several command line
    programs for job submission
    n   globus-job-run: Interactive jobs
    n   globus-job-submit: Batch/offline jobs
    n   globusrun: Flexible scripting infrastructure
n   Other High Level Interfaces
    n   General purpose
         n   Nimrod-G, Condor-G, PBS, GRD, etc
    n   Application specific
         n   ECCE’, Cactus, Web portals
n   For running of interactive jobs
n   Additional functionality beyond rsh
    n   Ex: Run 2 process job w/ executable staging
        globus-job-run -: host –np 2 –s myprog arg1 arg2
    n   Ex: Run 5 processes across 2 hosts
        globus-job-run \
                 -: host1 –np 2 –s myprog.linux arg1 \
                 -: host2 –np 3 –s myprog.aix arg2
    n   For list of arguments run:
        globus-job-run -help

n   For running of batch/offline jobs
    n   globus-job-submit                Submit job
        n   Same interface as globus-job-run
        n   Returns immediately
    n   globus-job-status          Check job
    n   globus-job-cancel     Cancel job
    n   globus-job-get-output Get job stdout/err
    n   globus-job-clean      Cleanup after job
n   Flexible job submission for scripting
    n   Uses an RSL string to specify job request
    n   Contains an embedded globus-gass-server
         n   Defines GASS URL prefix in RSL substitution variable:
    n   Supports both interactive and offline jobs
n   Complex to use
    n   Must write RSL by hand
    n   Must understand its esoteric features
    n   Generally you should use globus-job-* commands
                          Resource Brokers

         “Run a                                         “Supercomputers providing
distributed interactive        DIS-Specific       100 GFLOPS, 100 GB, < 100 msec latency”
 simulation involving            Broker
   100,000 entities”
                                                                   Supercomputer       Information
       “Create a               Collaborative                       resource broker       Service
 shared virtual space      environment-specific         " . . ."
   with participants         resource broker
     X, Y, and Z”                                                          “80 nodes on Argonne SP,
                                                                          256 nodes on CIT Exemplar
       “Perform                                                           300 nodes on NCSA O2000”
  a parameter study          Parameter study
                              specific broker        " . . ."
   involving 10,000                                                 Simultaneous
    separate trials”                                                    start
                             "Run SF-Express         "Run SF-Express        “Run SF-Express
                               on 80 nodes”           on 256 nodes”          on 300 nodes”
                                          Argonne                       CIT             NCSA
                                          Resource                    Resource         Resource
                                          Manager                     Manager          Manager
    Brokering via Lowering

n Resource location by refining a RSL
  expression (RSL lowering):
(MFLOPS=1000) Þ
 (& (arch=sp2)(count=200)) Þ
  (+ (& (arch=sp2) (count=120)

      (& (arch=sp2) (count=80)
    Remote I/O and Staging

n   Tell GRAM to pull executable from
    remote location
n   Access files from a remote location
n   stdin/stdout/stderr from a remote
              What is GASS?

(a) GASS file access API
  n   Replace open/close with
      globus_gass_open/close; read/write calls can
      then proceed directly
(b) RSL extensions
  n   URLs used to name executables, stdout,
(c) Remote cache management utility
(d) Low-level APIs for specialized behaviors
                GASS Architecture
 main( ) {
   fd = globus_gass_open(…)             (b) RSL extensions
   …                          GRAM             GASS Server
                                               HTTP Server
 (a) GASS file access API
                                                FTP Server
                                       (c) Remote cache
(d) Low-level APIs for                 management
customizing cache & GASS server         % globus-gass-cache
           GASS File Naming

n   URL encoding of resource names

    protocol       server address                 file name
n   Other examples
n   Supports http & https
n   Support ftp & gsiftp.
      GASS RSL Extensions

n   executable, stdin, stdout, stderr can
    be local files or URLs
n   executable and stdin loaded into local
    cache before job begins (on front-end
n   stdout, stderr handled via GASS
    append mode
n   Cache cleaned after job completes
     GASS/RSL Example

Example GASS Applications

n   On-demand, transparent loading of
    data sets
n   Caching of data sets
n   Automatic staging of code and data to
    remote supercomputers
n   (Near) real-time logging of application
    output to remote server
        GASS File Access API

n   Minimum changes to application
n    globus_gass_open(),
    n   Same as open(), close() but use URLs
        instead of filenames
    n   Caches URL in case of multiple opens
    n   Return descriptors to files in local cache
        or sockets to remote server
n    globus_gass_fopen(),
GASS File Access API (cont)

n   Support for different access patterns
    n   Read-only (from local cache)
    n   Write-only (to local cache)
    n   Read-write (to/from local cache)
    n   Write-only, append (to remote server)

URL in cache?    no
                      Download File
       yes             into cache

  open cached file,
     add cache
     reference                               globus_gass_close()

                                Modified    no Remove cache
                                    yes          reference

                                Upload changes
    GASS File API Semantics

n   Copy-on-open to cache if not truncate or
    write-only append and not already in cache
n   Copy on close from cache if not read only
    and not other copies open
n   Multiple globus_gass_open() calls share local
    copy of file
n   Append to remote file if write only append:
    e.g., for stdout and stderr
n   Reference counting keeps track of open files

n   Simple file server
    n   Run by user wherever necessary
    n   Secure https protocol, using GSI
    n   APIs for embedding server into other programs
n   Example
        globus-gass-server –r –w -t
    n   -r: Allow files to be read from this server
    n   -w: Allow files to be written to this server
    n   -t: Tilde expand (~/… à $(HOME)/…)
    n   -help: For list of all options
        GRAM & GASS: Putting It
 1.   Derive Contact String
 2.   Build RSL string
 3.   Startup GASS server
 4.   Submit to request                          program
 5.   Return output
                                            5     5   4
                          GASS server
 stdout                      3
 Host       1       Contact                     jobmanager
 name                string
Command         2        RSL
Line Args               string
    Example: A Simple Broker

n   Select machines based on availability
    n   Use MDS queries to get current host loads
    n   Look at output and figure out what
        machines to use
n   Generate RSL based on selection
    n   globus-job-run -dumprsl can assist
n   Execute globusrun, feeding it the RSL
    generated in previous step
           GRAM & GASS
n   Using RSL with globusrun
n   Running globus-gass-server
n   Modifying a program to use
    globus_gass_open() to read files
    remotely from a GASS server
 Globus Components In Action
  Local Machine
                                                        User                                  X509
      mpirun                                    RSL                   grid-proxy-init
                                 Machines               Proxy                                 User
                                                         Cert                                 Cert
           RSL multi-request

      globusrun     RSL parser               DUROC                    RSL single request

        GRAM                                                         GRAM
                  GSI                        GASS Server                         GSI
        Client                                                       Client

GRAM Gatekeeper          GRAM Job Manager               GRAM Gatekeeper             GRAM Job Manager

            GSI                GASS Client                            GSI                  GASS Client
                               PBS                                                       Unix Fork

                   App                                                        App
                               Nexus                                                        Nexus

                        AIX      MPI                                           Solaris       MPI
Remote Machine                                             Remote Machine
                     GRAM Components
                             MDS client API calls
                             to locate resources
           Client                                    MDS: Grid Index Info Server
                             MDS client API calls                                   Site boundary
                             to get resource info

 GRAM client API calls to
request resource allocation
                                               MDS:   Grid Resource Info Server
   and process creation.                                           Query current status
                          GRAM client API state                    of resource
        Globus Security     change callbacks
                                                    Local Resource Manager
                                                                                Allocate &
                                                                             create processes
                         Create      Job Manager

        Gatekeeper                  Parse
                                                       Monitor &
                                                        control       Process
                                      RSL Library
MDS: Monitoring and Discovery Service

  n   Learn how to use the MDS to locate
      and determine characteristics of
  n   Locate resources
      n   Where are resources with required
          architecture, installed software, available
          capacity, network bandwidth, etc.?
  n   Determine resource characteristics
      n   What are the physical characteristics,
          connectivity, capabilities of a resource?
    The Need for Information

n   System information is critical to operation of
    the grid and construction of applications
     n   How does an application determine what
         resources are available?
     n   What is the “state” of the computational grid?
     n   How can we optimize an application based on
         configuration of the underlying system?
n   We need a general information
    infrastructure to answer these questions
             Using Information for
              Resource Brokering
 “10 GFlops, EOS data,                                    Info service:
20 Mb/sec -- for 20 mins”                               location + selection
                              Resource   “What computers?”   Directory
                               Broker    “What speed?”        Service
               “20 Mb/sec”               “When available?”

                 Globus Resource
               Allocation Managers        “50 processors + storage
                                           from 10:20 to 10:40 pm”

    GRAM                    GRAM                 GRAM
Examples of Useful Information

  n   Characteristics of a compute resource
      n   IP address, software available, system
          administrator, networks connected to, OS
          version, load
  n   Characteristics of a network
      n   Bandwidth and latency, protocols, logical
  n   Characteristics of the Globus
      n   Hosts, resource managers
    Grid Information Service

n   Provide access to static and dynamic
    information regarding system components
n   A basis for configuration and adaptation in
    heterogeneous, dynamic environments
n   Requirements and characteristics
    n   Uniform, flexible access to information
    n   Scalable, efficient access to dynamic data
    n   Access to multiple information sources
    n   Decentralized maintenance

n   Store information in a distributed directories
    n   Directory stored in collection of LDAP servers
    n   Each server optimized for particular function
n   Directory can be updated by
    n   Information providers and tools
    n   Applications (i.e., users)
    n   Backend tools which generate info on demand
n   Information dynamically available to
    n   Tools
    n   Applications
Directory Service Functions
n   White Pages
    n   Look up the IP number, amount of memory, etc.,
        associated with a particular machine
n   Yellow Pages
    n   Find all the computers of a particular class or with
        a particular property
n   Temporary inconsistencies are often
    considered okay
    n   In a distributed system, you often do not know the
        state of a resource until you actually use it
    n   Information is often used as “hints”
    n   Information itself can contain ttl, etc.
              MDS Approach

n   Based on LDAP                        Application

    n   Lightweight Directory Access Protocol v3
        (LDAPv3)                       Middleware
    n   Standard data model
    n   Standard query protocol         LDAP API
n   Globus specific schema
    n   Host-centric representation
                                       GRIS    …     GIIS
n   Globus specific tools
    n   GRIS, GIIS
                                      SNMP         NWS          …
    n   Data discovery, publication,…
                                             NIS         LDAP
             MDS Components

n   Uses standard LDAP servers
    n   OpenLDAP, Netscape, Oracle, etc
n   Tools for populating & maintaining MDS
    n   Integrated with Globus Toolkit server release, not
        of concern to most Globus users
    n   Discover/update static and dynamic info
n   APIs for accessing & updating MDS contents
    n   C, Java, PERL (LDAP API, JNDI)
n   Various tools for manipulating MDS contents
    n   Command line tools, Shell scripts & GUIs
Anonymous Grid info search
n   grid-info-search -x -h
    Mds-Computer-isa: i686
    Mds-Computer-platform: i686
    Mds-Computer-Total-nodeCount: 1
    Mds-Cpu-Cache-l2kB: 512
    Mds-Cpu-features: fpu vme de pse tsc msr pae mce cx8 apic
       sep mtrr pge mca cmo
    v pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
    Mds-Cpu-Free-15minX100: 400
    Mds-Cpu-Free-1minX100: 400
    Mds-Cpu-Free-5minX100: 400
    Mds-Cpu-model: Intel(R) Xeon(TM) CPU 2

n   MDS provides the information needed
    to perform dynamic resource discovery
    and configuration
    n   Critical component of resource brokers
n   MDS is base on existing directory
    service standards (LDAPv3)

Shared By: