Docstoc

Slide presentation _power point 6.49MB_ - Digital Technology Center-ag

Document Sample
Slide presentation _power point 6.49MB_ - Digital Technology Center-ag Powered By Docstoc
					       Grid Computing and the
    Open Grid Service Architecture

                              Ian Foster
                       Argonne National Laboratory
                           University of Chicago


                     http://www.mcs.anl.gov/~foster




2nd IEEE Intl Symp. on Network Computing & Applications, Boston, April 17, 2003
                                                        3

            Partial Acknowledgements
      Open Grid Services Architecture design
       – Carl Kesselman, Karl Czajkowski @ USC/ISI
       – Steve Tuecke @ANL
       – Jeff Nick, Steve Graham, Jeff Frey @ IBM
      Grid services collaborators at ANL
       – Kate Keahey, Gregor von Laszewski
       – Thomas Sandholm, Jarek Gawor, John Bresnahan
      Globus Toolkit R&D also involves many fine
       scientists & engineers at ANL, USC/ISI, and
       elsewhere (see www.globus.org)
      Strong links with many EU, UK, US Grid projects
      Support from DOE, NASA, NSF, IBM, Microsoft
foster@mcs.anl.gov                       ARGONNE  CHICAGO
                                                        4



                       Overview
        Grid: why and what
        Evolution of Grid technology
         – Open Grid Services Architecture
        Future directions
         – Towards lightweight VOs: dynamic trust
           relationships
         – Towards global knowledge communities:
           virtual data and dynamic workspaces



foster@mcs.anl.gov                       ARGONNE  CHICAGO
                                                          5

                  Why the Grid?
             (1) Revolution in Science
     Pre-Internet
      – Theorize &/or experiment, alone
        or in small teams; publish paper
     Post-Internet
      – Construct and mine large databases of
        observational or simulation data
      – Develop simulations & analyses
      – Access specialized devices remotely
      – Exchange information within
        distributed multidisciplinary teams

foster@mcs.anl.gov                         ARGONNE  CHICAGO
                                                            6

                   Why the Grid?
             (2) Revolution in Business
     Pre-Internet
      – Central data processing facility
     Post-Internet
      – Enterprise computing is highly distributed,
        heterogeneous, inter-enterprise (B2B)
      – Business processes increasingly
        computing- & data-rich
      – Outsourcing becomes feasible =>
        service providers of various sorts


foster@mcs.anl.gov                           ARGONNE  CHICAGO
                                                                7

           New Opportunities
         Demand New Technology
   “Resource sharing & coordinated
    problem solving in dynamic, multi-
    institutional virtual organizations”




  “When the network is as fast as the computer's internal
   links, the machine disintegrates across the net into a set
   of special purpose appliances”   (George Gilder)
foster@mcs.anl.gov                           ARGONNE  CHICAGO
                                                           8

      Grid Communities & Technologies
     Yesterday
      – Small, static communities, primarily in science
      – Focus on sharing of computing resources
      – Globus Toolkit as technology base
     Today
      – Larger communities in science; early industry
      – Focused on sharing of data and computing
      – Open Grid Services Architecture emerging
     Tomorrow
      – Large, dynamic, diverse communities that share
        a wide variety of services, resources, data
      – New issues: Trust, distributed RM, knowledge
foster@mcs.anl.gov                          ARGONNE  CHICAGO
                                                    9



                      NSF TeraGrid
    NCSA, SDSC, Argonne, Caltech
    Unprecedented capability
     – 13.6 trillion flop/s
     – 600 terabytes of data
     – 40 gigabits per second
     – Accessible to thousands
       of scientists working on
       advanced research
    www.teragrid.org


foster@mcs.anl.gov                   ARGONNE  CHICAGO
                                    10




foster@mcs.anl.gov   ARGONNE  CHICAGO
    Data Grids for High
                                                11


      Energy Physics
   Enable international
    community of 1000s to
    access & analyze petabytes
    of data
   Harness computing &
    storage worldwide
   Virtual data concepts:
    manage programs,
    data, workflow
   Distributed system
    management
foster@mcs.anl.gov               ARGONNE  CHICAGO
      NEESgrid Earthquake Engineering
                                            12


               Collaboratory




                                U.Nevada Reno

                              www.neesgrid.org




foster@mcs.anl.gov           ARGONNE  CHICAGO
                                                                                         13
                           Grid Computing




                     Grid Computing
                     By M. Mitchell Waldrop
                     May 2002

                     Hook enough computers together and what do you get? A new kind of
                     utility that offers supercomputer processing on tap.

                     Is Internet history about to repeat itself?
foster@mcs.anl.gov                                       ARGONNE  CHICAGO
                                                                                                                                                          14

                                            Industrial Perspective on Grids:
                                             A Wide Range of Applications
                               Unique by Industry with Common Characteristics
    Grid Services Market Opportunity 2005




                                                                        Manufacturing
                                                          Financial
                                                          Services        Mechanical/        LS /
                                                                           Electronic   Bioinformatics
                                                                            Design                                                 Other
                                              Energy      Derivatives
                                                           Analysis        Process                         Entertainment
                                                                                            Cancer                                 Web
                                              Seismic                     Simulation       Research                             Applications
                                                          Statistical
                                              Analysis     Analysis         Finite           Drug              Digital            Weather
                                              Reservoir                    Element         Discovery          Rendering           Analysis
                                                           Portfolio       Analysis
                                              Analysis       Risk
                                                                                            Protein           Massive
                                                           Analysis                                          Multi-Player          Code
                                                                            Failure         Folding                              Breaking/
                                                                           Analysis                            Games
                                                                                                                                 Simulation
                                                                                            Protein
                                                                                          Sequencing          Streaming          Academic
                                                                                                                Media



                                                                        “Gridified” Infrastructure

                                                                                   Sources: IDC, 2000 and Bear Stearns- Internet 3.0 - 5/01 Analysis by SAI




foster@mcs.anl.gov                                                                                                 ARGONNE  CHICAGO
                                                        15



                       Overview
        Grid: why and what
        Evolution of Grid technology
         – Open Grid Services Architecture
        Future directions
         – Towards lightweight VOs: dynamic trust
           relationships
         – Towards global knowledge communities:
           virtual data and dynamic workspaces



foster@mcs.anl.gov                       ARGONNE  CHICAGO
                                                                       16



         Open Grid Services Architecture
        Service-oriented architecture
         – Key to virtualization, discovery,
           composition, local-remote transparency
        Leverage industry standards
         – Internet, Web services
        Distributed service management
         – A “component model for Web services”
        A framework for the definition of
         composable, interoperable services
     “The Physiology of the Grid: An Open Grid Services Architecture for
   Distributed Systems Integration”, Foster, Kesselman, Nick, Tuecke, 2002
foster@mcs.anl.gov                                    ARGONNE  CHICAGO
                                                          17

                     Web Services
        XML-based distributed computing technology
        Web service = a server process that exposes
         typed ports to the network
        Described by the Web Services Description
         Language, an XML document that contains
         – Type of message(s) the service understands &
           types of responses & exceptions it returns
         – “Methods” bound together as “port types”
         – Port types bound to protocols as “ports”
        A WSDL document completely defines a
         service and how to access it
foster@mcs.anl.gov                        ARGONNE  CHICAGO
                                                           18



                     OGSA Structure
        A standard substrate: the Grid service
         – Standard interfaces and behaviors that
           address key distributed system issues
         – A refactoring and extension of the Globus
           Toolkit protocol suite
        … supports standard service specifications
         – Resource management, databases,
           workflow, security, diagnostics, etc., etc.
         – Target of current & planned GGF efforts
        … and arbitrary application-specific
         services based on these & other definitions
foster@mcs.anl.gov                          ARGONNE  CHICAGO
                                                                           19

      Open Grid Services Infrastructure
                      Client
       Introspection:
                            Lifetime management
       • What port types?
                            • Explicit destruction
       • What policy?
                            • Soft-state lifetime
       • What state?
                     GridService    Data
                     (required)    access    Other standard interfaces:
Grid Service                                                    factory,
  Handle                                                   notification,
                      Service      Service     Service      collections
      handle            data         data        data
                      element      element     element
      resolution

Grid Service
 Reference
                            Implementation


                     Hosting environment/runtime
                          (“C”, J2EE, .NET, …)
foster@mcs.anl.gov                                   ARGONNE  CHICAGO
                                                                        20



      Open Grid Services Infrastructure

 GWD-R (draft-ggf-ogsi- gridservice-23)     Editors:
 Open Grid Services Infrastructure (OGSI)   S. Tuecke, ANL
 http://www.ggf.org/ogsi-wg                 K. Czajkowski, USC/ISI
                                            I. Foster, ANL
                                            J. Frey, IBM
                                            S. Graham, IBM
                                            C. Kesselman, USC/ISI
                                            D. Snelling, Fujitsu Labs
                                            P. Vanderbilt, NASA
                                            February 17, 2003

             Open Grid Services Infrastructure (OGSI)


foster@mcs.anl.gov                                ARGONNE  CHICAGO
                                                                           21

                      Example:
            Reliable File Transfer Service

                               Client          Client             Client


                         Request and manage file transfer operations
                                Grid     File   Notf’n Policy
   Fault                       Service Transfer Source
  Monitor                        Pending
               Query &/or                            interfaces
                subscribe      Performance
                                               service
             to service data     Policy        data
                                                          Internal
   Perf.                                       elements    State
  Monitor                        Faults


                                        Data transfer operations
foster@mcs.anl.gov                                  ARGONNE  CHICAGO
                                                           22

        Open Grid Service Architecture:
                  Next Steps
      Technical specifications
         – Open Grid Services Infrastructure is complete
         – Security, data access, Java binding, common
           resource models, etc., etc., in the pipeline
      Implementations and compliant products
         – Here: OGSA-based Globus Toolkit v3, …
         – Announced: IBM, Avaki, Platform, Sun, NEC,
           HP, Oracle, UD, Entropia, Insors, …, …
      Rich set of service defns & implementations


foster@mcs.anl.gov                        ARGONNE  CHICAGO
                                                          23

            Globus Toolkit v3 (GT3)
         Open Source OGSA Technology
        Implements OGSI interfaces
        Supports primary GT2 interfaces
         – High degree of backward compatibility
        Multiple platforms & hosting environments
         – J2EE, Java, C, .NET, Python
        New services
         – SLA negotiation, service registry, community
           authorization, data management, …
        Rapidly growing adoption and contributions:
         “Linux for the Grid”
foster@mcs.anl.gov                        ARGONNE  CHICAGO
                                                        24



                       Overview
        Grid: why and what
        Evolution of Grid technology
         – Open Grid Services Architecture
        Future directions
         – Towards lightweight VOs: dynamic trust
           relationships
         – Towards global knowledge communities:
           virtual data and dynamic workspaces



foster@mcs.anl.gov                       ARGONNE  CHICAGO
                                                          25


                     Future Directions
        Grids are about computers, certainly
         – “On-demand” access to computing, etc.
         – Challenging future issues here: e.g., scale




foster@mcs.anl.gov                         ARGONNE  CHICAGO
                                                                 26



         CMS Event Simulation Production
        Production Run on the Integration Testbed
          – Simulate 1.5 million full CMS events for physics
            studies: ~500 sec per event on 850 MHz processor
          – 2 months continuous running across 5 testbed sites
          – Managed by a single person at the US-CMS Tier 1




foster@mcs.anl.gov                               ARGONNE  CHICAGO
                                                                 27



         CMS Event Simulation Production
        Production Run on the Integration Testbed
          – Simulate 1.5 million full CMS events for physics
            studies: ~500 sec per event on 850 MHz processor
          – 2 months continuous running across 5 testbed sites
          – Managed by a single person at the US-CMS Tier 1




foster@mcs.anl.gov                               ARGONNE  CHICAGO
                                                          28


                     Future Directions
        Grids are about computers, certainly
         – “On-demand” access to computing, etc.
         – Challenging future issues here: e.g., scale
        But they are ultimately about people, their
         activities, and their interactions
         – New interaction modalities supported by on-
           demand formation of lightweight VOs
         – New technologies needed: e.g., trust,
           security, data and knowledge integration
        Convergence of interest between
         “Compute” and “Collaboration” Grids?
foster@mcs.anl.gov                         ARGONNE  CHICAGO
                                            29

        Global Knowledge Communities




foster@mcs.anl.gov           ARGONNE  CHICAGO
                                                         30



     Example Issue: Trust and Security
        Effective VO operation depends critically on
         – Trust: can I rely on you?
         – Protection mechanisms to govern actions
        Suffers from VO-organization policy mismatch
        Goal: collaborations no longer defined by slow
         centralized mechanisms but can
         – form spontaneously;
         – be managed in a distributed manner; and
         – be protected by an infrastructure that
           maintains and enforces trust relationships
foster@mcs.anl.gov                        ARGONNE  CHICAGO
                                                                                                                                                   31
                                Grid Security Services
                                     Requestor's                                                Service Provider's
                                       Domain                                                        Domain

                                      Trust                                                                 Trust
                                     Service                                                               Service
                    Attribute                      Authorization                       Authorization                     Attribute
                     Service                          Service                             Service                         Service



              Audit/                                     Privacy                      Privacy                                  Audit/
          Secure-Logging                                 Service                      Service                              Secure-Logging
              Service                                                                                                          Service



       Credential                                                                                                                     Credential
       Validation                                                                                                                     Validation
        Service                                                                                                                        Service

                                                                        Bridge/
                                                                      Translation
                                                                        Service

                                                                                                                                  Service
           Requestor
                                WS-Stub                       Secure Conversation                            WS-Stub              Provider
           Application
                                                                                                                                 Application




          Credential                                                                                                               Credential
          Validation                                                                                                               Validation
           Service                                                                                                                  Service
                                                      Authorization                 Authorization
                                                         Service                       Service
                         Attribute                                                                                   Attribute
                          Service                                                                                     Service
                                          Trust                                                         Trust
                                         Service                                                       Service
                                                                         VO
                                                                       Domain

foster@mcs.anl.gov                                                                                               ARGONNE  CHICAGO
                                                                                  32

           Understanding and Enhancing
              VO Trust and Security
   Usability
                     Community
   analysis
                                                               Establishment,
                                Social network analysis        enhancement,
                                Other analyses                  maintenance,
                                                                   verification
                     Trust    trust ( Tr, Te, As, L ) <- Cs;
   Monitoring for       recommend ( Rr, Re, As, L ) <- Cs;
     reputation,               Workflow analysis
    compliance,                Risk analysis
      intrusion
   detection, etc.   Policy            allowed (S, O, A, C)       Feasibility
                              Factoring wrt environment         analysis wrt
                                                                cost, legality,
                                                                etc.
                     Mechanism        VOTA, PKI, VPN, etc.


foster@mcs.anl.gov                                        ARGONNE  CHICAGO
                     Virtual Data
                                                           33


              for Collaborative Science
        Much collaboration is concerned with the
         development & use of knowledge, whether
         – Programs for data analysis and generation
         – Computations involving those programs
         – Metadata concerning data, programs,
           computations—and their interrelationships
        In a distributed, heterogeneous, fractal (?)
         environment with widely varying
         – Data and analysis program formats
         – Degrees of formality and scale
         – Scientific goals and sharing policies
foster@mcs.anl.gov                          ARGONNE  CHICAGO
              Sloan Digital Sky Survey
                                                 34


                 Production System




foster@mcs.anl.gov                ARGONNE  CHICAGO
                                                           35

                 Virtual Data Concept
        Capture and manage information about
         relationships among
         – Data (of widely varying representations)
         – Programs (& their execution needs)
         – Computations (& execution environments)
        Apply this information to, e.g.
         – Discovery: Data and program discovery
         – Workflow: Structured paradigm for organizing,
           locating, specifying, & requesting data
         – Explanation: provenance
         – Planning and scheduling
         – Other uses we haven’t thought of
foster@mcs.anl.gov                         ARGONNE  CHICAGO
                                                                    36
“I’ve come across some
interesting data, but I need
to understand the nature of
                                          Motivations
the corrections applied                  “I’ve detected a calibration
when it was constructed        Data      error in an instrument and
before I can trust it for my             want to know which derived
purposes.”                                   data to recompute.”

               created-by                consumed-by/
                                         generated-by



     Transformation       execution-of     Derivation
                                                 “I want to apply an
“I want to search an astronomical
                                              astronomical analysis
database for galaxies with certain
characteristics. If a program that            program to millions of
performs this analysis exists, I               objects. If the results
won’t have to write one from                  already exist, I’ll save
scratch.”
foster@mcs.anl.gov                           weeks of computation.”
                                                  ARGONNE  CHICAGO
                     Example:                                                                         37

            Sloan Galaxy Cluster Analysis

                                                                            DAG



                                           Sloan Data




                                           Galaxy cluster
                                  100000
                                           size distribution
                                  10000
             Number of Clusters




                                   1000



                                    100



                                     10
                                                                            Jim Annis, Steve Kent, Vijay
                                      1
                                           1            10            100
                                                                                Sehkri, Fermilab; Michael
foster@mcs.anl.gov                               Number of Galaxies
                                                                                   ARGONNE  CHICAGO
                                                                            Milligan, Yong Zhao, Chicago
                                                                                             38


              Integrating Provenance Data
                                       Group Index

                                                          Personal
                                                            VDS

                                                            DV
                                                                          Personal
                                                     DS
                                                                           Index
                          TR
                                                            DV                Personal
                                             DV
    Collaboration-                                                              Index
                          TR                                         DS
         level                                              DS
                                                                                  Personal
                                             TR
        index             TR                                DV                     Index
                     Collaboration       Group VDS          DV
                          VDS
                                                            DV

                                                          Personal
                                                            VDS




                                     Collaboration-wide
                                           index


foster@mcs.anl.gov                                                   ARGONNE  CHICAGO
                                                           39

                       Summary
     Yesterday
      – Small, static communities, primarily in science
      – Focus on sharing of computing resources
      – Globus Toolkit as technology base
     Today
      – Larger communities in science; early industry
      – Focused on sharing of data and computing
      – Open Grid Services Architecture emerging
     Tomorrow
      – Large, dynamic, diverse communities that share
        a wide variety of services, resources, data
      – New issues: Trust, distributed RM, knowledge
foster@mcs.anl.gov                          ARGONNE  CHICAGO
                                                 40



               For More Information
   The Globus Project™
    – www.globus.org
   Technical articles
    – www.mcs.anl.gov/~foster
   Open Grid Services Arch.
    – www.globus.org/ogsa
   Chimera
    – www.griphyn.org/chimera
   Global Grid Forum
    – www.gridforum.org
foster@mcs.anl.gov                ARGONNE  CHICAGO

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:0
posted:5/10/2013
language:English
pages:39
yaofenji yaofenji
About