Globus Project Future Directions

Document Sample
Globus Project Future Directions Powered By Docstoc
					   Enabling eScience:
    Open Software,
Standards, Infrastructure
                 Ian Foster
          Argonne National Laboratory
              University of Chicago
                 Globus Alliance

             www.mcs.anl.gov/~foster
UK eScience Meeting, Nottingham, September 2, 2004
The Grid Meets the BBC




 “The Grid is an international
  project that looks in detail at
  a terrorist cell operating
  on a global level and a team
  of American and British
  counter-terrorists who are
  tasked to stop it”

 Gareth Neame, BBC's head of drama
A Better Characterization?




  “The Grid is an international
   project that looks in detail at
   scientific collaborations
   operating on a global level
   and a team of computer
   scientists who are tasked
   to enable them”

  But perhaps not as telegenic?
                                                  4


     eScience & Grid: 6 Theses
1. Scientific progress depends increasingly on
   large-scale distributed collaborative work
2. Such distributed collaborative work raises
   challenging problems of broad importance
3. Any effective attack on those problems must
   involve close engagement with applications
4. Open software & standards are key to
   producing & disseminating required solutions
5. Shared software & service infrastructure are
   essential application enablers
6. A cross-disciplinary community of technology
   producers & consumers is needed
                                                       5


     Implication: A Problem-Driven,
     Collaborative R&D Methodology
                                             Infra-
Software,                                  structure
Standards
              Build             Deploy
                                Deploy



                                   Apply
                                   Apply
            Design                Apply
                                  Apply


Computer              Analyze              Discipline
 Science                                   Advances
                                            6



                     Overview
   How are we doing?
       Software
       Standards
       Infrastructure
       Community
   An advertorial, and request for input
       Globus Toolkit version 4
   Summary
                                            7



                     Overview
   How are we doing?
       Software
       Standards
       Infrastructure
       Community
   An advertorial, and request for input
       Globus Toolkit version 4
   Summary
                                                  8



        Why Open Software Matters
   eScience requires sophisticated
    functionality but is a small “market”
   Commercial software does not meet needs
   Open software can help jumpstart
    development by reducing barriers to entry
       Encourage adoption of common approaches
        to key technical problems
       Enable broad Grid technology ecosystem
   A basis for international cooperation
   A basis for cooperation with industry
                                                       9

              “Open Software” is
         Ultimately about Community
   Contributors: design, development, packaging,
    testing, documentation, training, support
       United by common architectural perspective
   Users
       May be major contributors via, e.g., testing
   Governance structure
       To determine how the software evolves
   Processes for coordinating all these activities
       Packaging, testing, reporting, …
   An ecosystem of complementary components
       Enabled by appropriately open architecture
                                           10


             “Ecosystem”?
Not a monoculture …   … or Cambrian explosion




          … but a web of components
                                                      11

     E.g., Globus Alliance & Toolkit
(Argonne, USC/ISI, Edinburgh, PDC, NCSA)
   An international partnership dedicated to
    creating & disseminating high-quality open
    source Grid technology: the Globus Toolkit
       Design, engineering, support, governance
   Academic Affiliates make major contributions
       EU: CERN, MPI, Poznan, INFN, etc.
       AP: AIST, TIT, Monash, etc.
       US: SDSC, TACC, UCSB, UW, etc.
   Significant industrial contributions & adoption
   1000s of users worldwide, many contribute
                                                      12

       Broader Ecosystem*:
  Example Complementary Projects
    NSF Middleware Initiative
        Packaging, testing, additional components
    Virtual Data Toolkit (GriPhyN + PPDG)
        GT, Condor, Virtual Data System, etc.
    EGEE and “gLite”
        Close collaboration with Globus + Condor
    TeraGrid, Earth System Grid, NEESgrid, …
        Consume and produce components
    Open Middleware Infrastructure Institute
        Collaboration on components, testing, etc.
* See tutorial by Lee Liming: AHM, GGF, SC’2004.
                                              13
       Broader Ecosystem:
E.g., NMI Distributed Test Facility




 (NSF Middleware Initiative’s GRIDS Center)
How Grid Software Works: NSF Network for
Earthquake Engineering Simulation (NEES)
 Transform our ability to carry out research vital to
 reducing vulnerability to catastrophic earthquakes
                                        15

       Building a NEES Collaboratory:
            What the User Wants
Secure,
reliable, on-
demand
access to
data,
software,
people, and
other
resources
(ideally all
via a Web
Browser)
                                                                                           16

                   How it Really Happens
                    (A Simplified View)
                                                                         Compute
                               Simulation                                 Server
                                  Tool                                   Compute
        Web
      Browser                                                             Server
                    Web                        Registration
                   Portal                        Service
                                                                          Camera

                                 Data         Telepresence
                                Viewer           Monitor                  Camera
                                 Tool
                                                                         Database
                            Chat
                                                                          service
                            Tool
                                                  Data
                                                                         Database
                     Credential                  Catalog
                                                                          service
                     Repository
                                                                         Database
                 Certificate                                              service
                 authority
Users work       Application services       Collective services    Resources implement
with client      organize VOs & enable        aggregate &/or       standard access &
applications    access to other services    virtualize resources   management interfaces
                                                                                               17

                     How it Really Happens
                    (without Grid Software)
                                                                                 Compute
                                                                            A
                                   Simulation                                     Server
                                      Tool                                       Compute
        Web                                                                 B
      Browser                                                                     Server
                        Web                        Registration
                       Portal                        Service
                                                                                Camera

                                     Data         Telepresence
Application
Developer
               10
                                    Viewer           Monitor                    Camera
Off the                              Tool
               12
Shelf                                                                            Database
                                Chat                                        C
Globus                                                                            service
Toolkit
               0                Tool
                                                      Data
                                                                                 Database
Grid                     Credential                  Catalog                D
Community
               0                                                                  service
                         Repository
                                                                                 Database
                     Certificate                                            E
                                                                                  service
                     authority
Users work           Application services       Collective services    Resources implement
with client          organize VOs & enable        aggregate &/or       standard access &
applications        access to other services    virtualize resources   management interfaces
                                                                                              18

                      How it Really Happens
                      (with Grid Software)
                                                                         Globus   Compute
                                                                         GRAM      Server
                                  Simulation
                                     Tool                                         Compute
        Web                                                              Globus
                                                                         GRAM      Server
      Browser
                                                  Globus Index
                      CHEF                           Service
                                                                             Camera

                                    Data         Telepresence
Application
Developer
               2
                                   Viewer           Monitor                  Camera
Off the                             Tool
               9
Shelf                                                                    Globus   Database
                          CHEF Chat                                       DAI
Globus                                                                             service
               5           Teamlet
Toolkit                                              Globus
                                                                         Globus   Database
Grid                                                MCS/RLS               DAI
               3         MyProxy                                                   service
Community
                                                                         Globus   Database
                    Certificate                                           DAI      service
                    Authority
Users work          Application services       Collective services    Resources implement
with client         organize VOs & enable        aggregate &/or       standard access &
applications       access to other services    virtualize resources   management interfaces
             Number of Participants




            10
                 20
                      30
                           40
                                50
                                         60
                                                             70




        0
 8:00
 8:30
 9:00
 9:30
10:00
10:30
11:00
11:30
12:00
12:30
13:00
                                                                     NEESgrid


13:30
14:00
14:30
                                                                    (July 2003)
                                                                  Multisite Online


15:00
                                                                  Simulation Test

15:30
16:00
16:30
17:00
17:30
                                                 UIUC


18:00
18:30
                                      Colorado




             Illinois
                                     Illinois (simulation)
                                                                                     19




            Colorado
                                                           20


              NEESgrid Summary
   A successful “turn of the crank”
       S/w produced & deployed on time & budget,
        and new applications enabled
       A producer as well as consumer of Grid s/w
   Many sociopolitical “learning opportunities”
       4 tasks: develop s/w, engineer s/w, elicit
        requirements, educate community
       Experiment-driven deployment™ was key
   “No victory is final”: challenges remain
       Hand off s/w to separate operations team
       Sharing of facilities, data: politically charged
                                                         21


             Software: Summary
   Good software arises from trying to solve real
    problems in real projects—& then generalizing
       E.g., Globus: security, job submission/mgmt,
        data movement, monitoring, etc.
   The result is solutions that make sense within
    a wide variety of applications
       Solve real problems, but not every problem
   Resulting software is not a “turnkey” solution
    for any significant application
       “Turnkey” solutions require integration
       Factoring can extract higher-level “solutions”
                                                22



              Example “Solutions”
   Portal-based User Registration System (PURSE)
       Source: Earth System Grid, PDC
       Web-based A&A management
   Lightweight Director Replicator
       Source: LIGO
       Data replication management
   Workflow execution & management
       DAGman + Condor-G + Globus components
       Source: Virtual data toolkit
   Service monitoring & fault detection
       Source: Earth System Grid
                                            23



                     Overview
   How are we doing?
       Software
       Standards
       Infrastructure
       Community
   An advertorial, and request for input
       Globus Toolkit version 4
   Summary
                                                      24

                “Standards”:
             Examples of Success
   Grid Security Infrastructure
       Broadly used, multiple implns, WS-Security
       Rich Grid security ecosystem, with linkages
        to MyProxy, OTP, KX509, Shibboleth, …
   GridFTP
       Broadly used, multiple implementations
   WSDL/SOAP
       Facilitating service-oriented architectures
   OGSI/WSRF
       Many find encode useful patterns, behaviors
                                                        25


               Standards: Status
   Open Grid Services Architecture (OGSA)—the
    lighthouse by which we steer
       Defines requirements & priorities
       But far from complete
   W3C, OASIS, GGF, DMTF, IETF
       Good things are happening in many areas
       WS-Agreement, DAIS, SRM, …, …
   But for those building systems today?
       Problem areas: monitoring, policy, data, etc.
       Ad hoc approaches: will cost us big later
 “Experiment-driven deployment” on intl. scale
 to drive interoperability of infrastructure, code
                                            26



                     Overview
   How are we doing?
       Software
       Standards
       Infrastructure
       Community
   An advertorial, and request for input
       Globus Toolkit version 4
   Summary
                                                      27



                  Infrastructure
   Broadly deployed services in support of
    virtual organization formation and operation
       Authentication, authorization, discovery, …
   Services, software, and policies enabling on-
    demand access to important resources
       Computers, databases, networks, storage,
        software services,…
   Operational support for 24x7 availability
   Integration with campus infrastructures
   Distributed, heterogeneous, instrumented
    systems can be wonderful CS testbeds
Grid2003: An Operational Grid                   28

 28 sites (2100-2800 CPUs) & growing
 400-1300 concurrent jobs
 8 substantial applications + CS experiments
 Running since October 2003




          Korea

              http://www.ivdgl.org/grid2003
                                              29

    Grid2003 Software Stack
     (“Virtual Data Toolkit”)
               Application
     Chimera Virtual Data System
        DAGMan and Condor-G
Globus Toolkit GSI, GRAM, GridFTP, etc.
    Site schedulers and file systems
     Clusters and storage systems
   Three levels of deployment:
   + Site services: GRAM, GridFTP, etc.
   + Global & virtual organization services
   + IGOC: iVDGL Grid Operations Center
                                                                  30



                  Grid2003 Metrics

            Metric                   Target Achieved
Number of CPUs                         400      2762 (28 sites)

Number of users                       > 10         102 (16)

Number of applications                 >4         10 (+CS)
Number of sites running concurrent
                                      > 10            17
apps

Peak number of concurrent jobs        1000           1100

Data transfer per day                > 2-3 TB    4.4 TB max
                                                  31


     Grid2003 Applications To Date

   CMS proton-proton collision simulation
   ATLAS proton-proton collision simulation
   LIGO gravitational wave search
   SDSS galaxy cluster detection
   ATLAS interactive analysis
   BTeV proton-antiproton collision simulation
   SnB biomolecular analysis
   GADU/Gnare genone analysis
   Various computer science experiments
       www.ivdgl.org/grid2003/applications
              Example Grid3 Application:                          32


               NVO Mosaic Construction
Construct custom mosaics on demand from multiple data sources
User specifies projection, coordinates, size, rotation, spatial
sampling




NVO/NASA Montage: A small (1200 node) workflow
Work by Ewa Deelman et al., USC/ISI and Caltech
                                                  33
               Next Step:
            Open Science Grid
   U.S. (international?) consortium to provide
    services to a broad set of sciences
   Grid3 as a starting point, expanding to
    include many more sites
   A major focus is the MOU/SLA structure
    required to sustain & scale operations
       Resource providers
       Resource consumers
       Virtual organizations
   We hope to collaborate with TeraGrid, EGEE,
    UK NGS, etc.
                                                       34



          Infrastructure: Summary
   Encouraging progress
       Real understanding of how to operate Grid
        infrastructures is emerging
       Production infrastructures are appearing
        and are being relied upon for real science
   Significant areas of concern remain
       Security is going to get harder
       International interoperability still elusive
       We haven’t got the right model for sustained
        infrastructure development & support
                                            35



                     Overview
   How are we doing?
       Software
       Standards
       Infrastructure
       Community
   An advertorial, and request for input
       Globus Toolkit version 4
   Summary
                                                      36

                   Community
   Big picture is extremely positive
       The “eScience”/“Grid” community is large,
        enthusiastic, smart, and diverse
       Significant exchange of ideas, software,
        personnel, experiences
       Real application-CS cooperation
   We can do better in various specific areas
       Not clear we’re always focusing on the real
        problems: often viewed as “mundane”??
       CS community could be even more engaged
       Software development a community effort
  Global
Community
                                            38



                     Overview
   How are we doing?
       Software
       Standards
       Infrastructure
       Community
   An advertorial, and request for input
       Globus Toolkit version 4
   Summary
                                                       39
              What’s New in
        GT 4.0 (January 31, 2005)
   For all:
       Additions: data, security, execution, XIO, …
       Improved packaging, testing, performance,
        usability, doc, standards compliance (phew)
       WS components ready for broader use
   For the end user:
       More complementary tools & solutions
       C, Java, Python APIs; command line tools
   For the developer:
       Java (Axis/Tomcat) hosting greatly improved
       Python (pyGlobus) hosting for the first time
40
                                                                  41
                         Apache Axis
                    Web Services Container
                  Good news for Java WS developers: GT4.0
                   works with standard Axis* and Tomcat*
GT      App
                      GT provides Axis-loadable libraries, handlers
bits    bits
                      Includes useful behaviors such as inspection,
                       notification, lifetime mgmt (WSRF)
                      Others implement GRAM, etc.
   Security
  Addressing      Major Globus contributions to Apache
   Axis
                      ~50% of WS-Addressing code
                      ~15% of WS-Security code
                      Many bug fixes
                      WSRF code a possible next contribution
                   * Modulo Axis and Tomcat release cycle issues
                                                     42



            Standards Compliance
   Web services: WS-I compliance
       All interfaces support WS-I Basic Profile,
        modulo use of WS-Addressing
   Security
    a) WS-I Basic Security Profile (plaintext)
    b) IETF RFC 3820 Proxy Certificate
   GridFTP
       GGF GFD 020
   Others in progress & being tracked
       WSRF (OASIS), WS-Addressing (W3C),
        OGSA-DAI (GGF), RLS (GGF)
                                                  43
         Globus Ecosystem
(Just a Few Examples Listed Here)
   Tools provide higher-level functionality
       Nimrod-G, MPICH-G2, Condor-G, Ninf-G
       NTCP telecontrol
       GT4IDE Eclipse IDE
   Packages integrate GT with other s/w
       VDT, NMI, CTSS, NEESgrid, ESG
   Solutions package a set of functionality
       VO management, monitoring, replica mgmt
   Documentation, e.g.
       Borja Sotomayor’s tutorial
                                                            44



           GT4.0 Release Schedule

Date       Stability        Features     Public interfaces
            Level         added after?    changed after?

Aug 3        Alpha            Yes                Yes

Oct 15   Full-featured        No           Yes, but only if
         development                     significant benefits
Dec 3    Beta-quality         No                 No
         development
 Jan     Stable release       No                 No
 31,        (FINAL)
2005
                                                        45


      We’d Getting a Lot of Help,
     But Could do with A Lot More
   Testing and feedback
       Users, developers, deployers: plan to use the
        software now & provide feedback
       Tell us what is missing, what performance
        you need, what interfaces & platforms, …
       Ideally, also offer to help meet needs (-:
   Related software, solutions, documentation
       Adapt your tools to use GT4
       Develop new GT4-based components
       Develop GT4-based solutions
       Develop documentation components
                                            46



                     Overview
   How are we doing?
       Software
       Standards
       Infrastructure
       Community
   An advertorial, and request for input
       Globus Toolkit version 4
   Summary
                                                  47


      eScience & Grid: 6 Theses
1. Scientific progress depends increasingly on
   large-scale distributed collaborative work
2. Such distributed collaborative work raises
   challenging problems of broad importance
3. Any effective attack on those problems must
   involve close engagement with applications
4. Open software & standards are key to
   achieving a critical mass of contributors
5. Shared software & service infrastructure are
   essential application enablers
6. A cross-disciplinary community of technology
   producers & consumers is vital
                                                    48



       Overall, We are Doing Well
   Communities & individuals are, increasingly,
    using the Grid to advance their science
   Broad consensus on many key architecture
    concepts, if not always their implementation
   Significant base of open source software,
    widely used in applications & infrastructure
   Service-oriented arch facilitates cooperation
    on software development & code reuse
   Grid standards are making a difference on a
    daily basis: e.g., GSI, GridFTP
                                                 49



    Overall, We are Doing Well (2)
   A real understanding of how to operate
    Grid infrastructures is emerging
   Production infrastructures are appearing
    and are being relied upon for real science
   Productive international cooperation is
    occurring at many levels
   A vibrant community has formed and
    shows no signs of slowing down
   Real connections have been formed
    between computer science & applications
                                                   50


            Problem-Driven,
     Collaborative R&D Methodology
                                             Infra-
Software,                                  structure
Standards
              Build             Deploy
                                Deploy

                        Global
                      Community
                                   Apply
                                   Apply
            Design                Apply
                                  Apply


Computer              Analyze              Discipline
 Science                                   Advances
                                           51


         Software Ecosystem
Not a monoculture …   … or Cambrian explosion




          … but a web of components
                                                         52



        We Can Certainly Do Better
   Be smarter about how we work with users
       Not enough to point people at a manual
   Treat s/w as shared infrastructure, to be
    developed, engineered, tested, improved
       Be honest about costs & time scales, expertise
   Establish real collaboration on software
       Partition the space of what to do: it’s large
       Partners, not customers or competitors
   Tackle process issues explicitly
       Standardize on packaging, testing, support
       Deployment, operations, security issues
                                                          53



    We Can Certainly Do Better (2)
   Aspire to code reuse & interoperability
       Interoperability layers are not the answer
       Recognize the costs of noninteroperability
   Focus standards efforts on the real problems
    faced when sharing software & infrastructure
       Quit fiddling with Web services infrastructure!
   Build sustained, critical mass teams
       Problems are hard; requires time & expertise
   Build and operate large-scale Grids with real
    application groups to drive all of this
       With explicit O(5) year focus and goals
                                                      54



         Thanks, in particular, to:
   Carl Kesselman and Steve Tuecke, my long-
    time Globus co-conspirators
   Kate Keahey, Lee Liming, Jennifer Schopf,
    Gregor von Laszewski, Mike Wilde @ Argonne
   Globus Alliance members at Argonne,
    U.Chicago, USC/ISI, Edinburgh, PDC, NCSA
   Miron Livny, U.Wisconsin Condor project
   Other partners in Grid technology, application,
    & infrastructure projects
   DOE, NSF (esp. NMI program), NASA, IBM,
    Microsoft for generous support
                                                      55


              For More Information
   Globus Alliance
       www.globus.org
   Global Grid Forum
       www.ggf.org
   Open Science Grid
       www.opensciencegrid.org
   Background information
       www.mcs.anl.gov/~foster
                                     2nd Edition
   GlobusWORLD 2005              www.mkp.com/grid2
       Feb 7-11, Boston
Extra Slides
                                                        57


    Globus Toolkit: A Brief History
   GT 1.0 (1998) to 2.0 (2002)
       PKI-based Grid Security Infrastructure
       Execution (GRAM), data (GridFTP), info (MDS)
       Gradual introduction of support processes
   GT 3.0 (June 2003) and 3.2 (Feb 2004)
       International collaboration
       Higher-level services: replica location, file
        transfer, registry, credential repository
       Refactoring GT mechanisms  WS framework
       Most production deployments recommended
        to use pre-WS (“GT2.4”) components
                                                       58


         NEESgrid Software Details
   Inputs
       Essentially all of GT3.2—GSI, GridFTP, GRAM,
        MDS, … (coherent architecture helps!)
       CHEF, Creare Data Turbine, OpenSEES
   Custom development
       NTCP telecontrol, data management, etc.
   Integration
       All of the above, and more
   Outputs
       NEESgrid system  NEES consortium
       NTCP components  Globus Toolkit  …

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:5
posted:8/14/2012
language:
pages:58