Docstoc

Resource Brokering_ the EUROGRIDGRIP Approach

Document Sample
Resource Brokering_ the EUROGRIDGRIP Approach Powered By Docstoc
					        Resource Brokering: the
        EuroGrid/GRIP approach


    Donal Fellows, John Brooke, Jon MacLaren
E-Science NorthWest @ University of Manchester UK
              http://www.esnw.ac.uk
                                                  Grid
                                        Interoperability
In European and Japanese Grid projects there are two major
middleware systems deployed, Globus (US) and Unicore
(Europe/Japan).

Globus is mainly deployed in cluster-based Grids and
Unicore in projects with complex heterogeneous
architectures (e.g. specialist HPC architectures).

The FP 5 project GRIP began looking at the question of how
resource requests could be handled from Unicore to Globus
and the FP6 project takes this work forward into the world of
service-based architectures (e.g. OGSA)

                                                                2
                                          Starting point - GRIP




EU Funded FP5 Project as part of Information Society Technologies Programme IST
2001-32257
http://www.grid-interoperability.org/
                                   A Dual Job-Space

Thus we have a space of “requests” defined as a vector
space of the computational needs of users over a Grid. For
many jobs most of the entries in the vector will be null.
We have another space of “services” who can produce “cost
vectors” for costing for the user jobs (providing they can
accommodate them).
This is an example of a dual vector space.
A strictly defined dual space is probably too rigid but can
provide a basis for simulations.
The abstract job requirements will need to be agreed. It may
be a task for a broker to translate a job specification to a
“user job” for a given Grid node.

                                                               4
                              4 - Dual Space

                                    Scalar cost
                                    in tokens

                 1

    Job vector
C
o                2
s
t                    Cost vector



     User Job




                                                  5
                                           Computational
                                               resource
Computational jobs ask questions about the internal
structure of the provider of computational power in a manner
that an electrically powered device does not.
For example, do we require specific compilers, libraries, disk
resource, visualization servers?
What if it goes wrong, do we get compensation? If we
transfer data and methods of analysis over the Internet is it
secure?


A resource broker for high performance computation is a
different order of complexity to a broker for an electricity
supplier.

                                                                 6
EuroGrid: Meteo-Grid




                   7
                               Resource Requestor
                               and Provider Spaces
Resource requestor space (RR), in terms of what the user
wants: e.g. Relocatable Weather Model, 10^6 points, 24
hours, full topography.
Resource Provider space (RP), 128 processors, Origin 3000
architecture, 40 Gigabytes Memory, 1000 Gigabytes disk
space, 100 Mb/s connection.
We may even forward on requests from one resource
provider to another, recasting of O3000 job in terms of IA64
cluster, gives different resource set.
Linkage and staging of different stages of workflow require
environmental support, a hosting environment.
We can have multiple offers in RP space for the same RR
values

                                                               8
                        Abstract Functions for a
                               resource broker
1. Resource discovery, for workflows as well as single
   jobs.
2. Resource capability checking, do the offering sites
   have ALL necessary capability and environmental
   support for instantiating the workflow.
3. Inclusion of Quality of Service policies in the offers.
4. Information necessary for the negotiation between
   client and provider and mechanisms for ensuring
   contract compliance.

Document submitted to GPA-RG group of GGF.


                                                             9
               Brokers as Virtual
                  Organizations

Users




Virtual
Organisation
Brokers

                             Organization
System                       Firewalls
Brokers




Compute
Resources



                                    10
                                                     Federated
                                                     Brokering
                                                          Resource


Client                 Broker          Broker
                       Service         Service            Resource
         Broker
Client   Service
                                                          Resource
                       Broker
Client                 Service
                                       Broker
         Broker                        Service            Resource
Client   Service
                       Broker          Replication
                       Service                            Resource
Client

                                       Broker
                                       Service            Resource


         VO Layer   Specialist Layer   Site Layer




                                                                 11
                                                                         Brokering and
                                                                        OGSA Services
Persistent Virtual Environments
                                                Clients                  Other Brokers



                                                                                          Banking Services


      Metascheduling
         Service                                Broker


                                                                              Site Feedback Policy Manager




    Workflow Manager              Chargeable Schedulable GridServices           Resource Usage Monitor




                                                                                                             12
                              Possible OGSA Broker

Interoperating OGSA services
                                     Resource     Resource
                                      Broker      Database




                                                         TSI
                                      Network
                           Unicore
          Unicore Client                Job         OGSA Server A
                           Gateway
                                     Supervisor
                                                        GT3/4



                                       User
                                     Database




                                                                13
                                                              Site Configuration
                                       Gateway
                                                   Users Contact NJSes or
                                                   Brok er (for site-wide brok ering)


      Delegate (site-wide brok ering only)                Delegate (site-wide brok ering only)




                                                                                          LRC
       LRC




NJS                                          Broker                                              NJS




             IDB                                                               IDB
                                         Potential to Share
                                         (Partial?) IDBs
                                         between NJSes
                                         (CSAR Config?)
                 TSI Supplies
                 Dynamic Data
TSI              to IDB                                                                          TSI

                                                                                                       14
                                                                               UoM Broker architecture
                                                                                                     To outside world

                                                                                                                                                   Look up signing identity
                                               IDB                                     NJS                                 UUDB
                                                                                                                                 Verify delegated
                                                      Look up                                Broker hosted in NJS
                                                                                                                                 identities
                                                      configuration
       Look up static                                                        AbstractBroker                                                          TicketManager
       resources                                                                                          Get & check signed tickets (contracts)



                                    SingleVsiteBroker                                                        HostVsite Map                        WholeUsiteBroker
                                                                                                                                                                    Use R-GMA to provide
             Delegate to Grid architecture-specific                                                                                                                 information for all Usite
             engine for local resource check                                        Delegate to application-domain expert code                                      component hosts

                               LocalResourceChecker                                       Experts may use LRC
                                                                                                                     ExpertBroker                          R-GMA

                                      Pass untranslatable
                                      resources to Unicore
                                      resource checker
              UnicoreRC                                        Globus2RC                Globus3RC                   DWDLMExpert                       ICMExpert                 Other
            Look up                      Look up resources
            dynamic                                                                                 Delegate resource domain translation
            resources
                                                                                                    Converts UNICORE resource requests to XPath
                    TSI        GRAM MDS                          SimpleTranslator                   search terms for GT3 Index Service & set of
                                                                                                    untranslatable resources to use UNCORE standard
                                                                                                    techniques upon.
                        Compute                  GT3            Look up resources
                                                                                                                             OntologicalTranslator
                        Resource                                                                                                                   Look up translations appropriate
                                                                                                                                                   to target Globus resource schema
Key:                                                                  SimpleTranslator converts delegated UNICORE
  UNICORE Components                                                  resource requests into LDAP search terms for
  EUROGRID Broker
                                                                      GT2 MDS & set of untranslatable resources to
                                                                      use UNICORE standard techniques upon.
                                                                                                                                       Ontology
  Globus Components
  GRIP Broker
  Whole-Site Broker
  Inheritance relation
                                                                                                                                                                                   15
                                Broker functions

• A simple Resource Check request: “Can this job
  run here”, checks static qualities like software
  resources (e.g. Gaussian98) as well as capacity
  resources like quotas (disk quotas, CPU, etc.)

• A Quality of Service request: returns a range of
  turnaround time, and cost, as part of a Ticket. If
  the Ticket is presented (within its lifetime) with the
  job, the turnaround and cost estimates should be
  met.



                                                       16
                                      Grid Resource
                                 Description Problem
Two Independent Grid Systems
      Unicore (http://www.unicore.org/)
      Globus (http://www.globus.org/)
Both Need to Describe Systems that run Compute Jobs
Very Different Description Languages
      Unicore’s Resource model, part of the AJO Framework
      Globus’s GLUE Schema (DataTAG, iVGDL) for GT2
      and GT3
For interoperability, we want to take a Unicore job and run it
  on Globus resources
Therefore, we need to translate the Job’s Resource
  Requirements between the two Systems


                                                                 17
                                    Methodology for
                                  translation servce
Address Data Transformation Issues for Translating
   Attributes
Find a technology that has these characteristics:
      can model the two ontologies
      has support for linking abstract concepts to code
      fragments
      easily allows someone to update mappings
      is appropriate for a video conferencing setting
      writes modelling information to a file format that
      can be used by other applications
Use the data files created by the application to run the
   translator service.


                                                           18
Unicore: Modelling
       Resources




                 19
             GLUE:
Modelling resources




                  20
GLUE: Marking up
      transcripts




                21
GLUE: Provenance
      Information




                22
                                        Compatible Concepts


Unicore Ontology                  GLUE Ontology
Network Performance               Glue SI00 Benchmark
Network Performance               Glue SF00 Benchmark
Floating Point Performance        Glue SI00 Benchmark
Floating Point Performance        Glue SF00 Benchmark
Data Processing Performance       Glue S100 Benchmark
Data Processing Performance       Glue SF00 Benchmark
Maximum Memory Capacity Request   Host Virtual Main Memory Available
Maximum Memory Capacity Request   Subcluster Virtual Main Memory Available
Minimum Memory Capacity Request   Host RAM Main Memory Available
Minimum Memory Capacity Request   Subcluster RAM Main Memory Available
Priority Value                    Priority




                                                                             23
Translation Service
          Prototype




                  24
                                             Conclusions

• Interoperability of grid resource requests is at the heart of
the abstract idea of computational resource that can cross
Grid domain boundaries
• We wish to provide application users with seamless access
to resources, they should not need to know details of the
machines on which they run.
• High level abstractions do not yet exist as standards, so we
have to create ontologies that can translate differing
modelling abstractions for Grid resources.
• Our current translations lose much information in crossing
between current middleware systems (e.g. Globus and
Unicore).

                                                              25
                                  Continuation of
                         interoperability research
Research Centre Jülich         http://www.unigrids.org
(Project manager)

Consorzio Interuniversitario
per il Calcolo Automatico
dell’Italia Nord Orientale

Fujitsu Laboratories of Europe

University of Warsaw

Intel GmbH

University of Manchester

T-Systems SfR

                                                         26
                                            GLUE: Container
                                                   Classes
GLUE has container classes that include “Computing Element”, “Cluster”,
  “Subcluster” and “Host”. From the heading “Representing Information”,
  the GLUE document indicates:
   “…hosts are composed into sub-clusters, sub-clusters are grouped
      into clusters, and then computing elements refer to one or more
      clusters.”




These container objects may hold any number optional auxiliary classes
that actually describe the GRID features.



                                                                          27
                                                                            GLUE: Auxiliary
                                                                                   Classes
The documentation provides few details about the nature of a Host other than that it is a
“physical computing element”. Much of the meaning for Host has to be derived from what it
might contain. Consider the following two valid definitions:




      A Host is a physical computing element characterized by
      Main Memory, a Benchmark, a Network Adapter and an        A Host is a physical computing element characterized by an
      Operating System                                          Architecture, a Processor and an Operating System.




                                                                                                                             28
                                                   Map concepts
                                              between ontologies




Unicore and GLUE have different philosophies for describing resources :-(
In Unicore, the resources are described in terms of resource requests
In GLUE, resources are described in terms of the availability of resources.




                                                                              29
                                        Local Brokering
                                         Configurations

  Client                                 Client


 Gateway                                Gateway


                                                              Broker
    NJS                Broker

                                 NJS      NJS        NJS      R-GMA
              IDB


TSI/Host                GT3      Host     Host       Host
Normal EUROGRID/GRIP Brokering          Site-Wide Brokering

                                                                       30
                                                    RR and RP Spaces
                                                request
                     RR space                                                   RP space




                                request
                                                                B
A
                                                                                    sync

                     RP space


                                     RR space                                 RP space


                          Request
    C                     referral
                                                            D




Figure 1: Request from RR space at A mapped into resource providers at B and C, with C
forwarding a request formulated in RR space to RP space at D. B and D synchronize at end of
workflow before results returned to the initiator A.



                                                                                              31

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:3
posted:8/10/2012
language:English
pages:31