Docstoc

Collaboration_ Grids_ and Virtual Organisations

Document Sample
Collaboration_ Grids_ and Virtual Organisations Powered By Docstoc
					                     Collaboration, Grids,
                          and Virtual
                         Organisations
                Stephen Pickles

                Reading, 21st March 2007




Combining the strengths of UMIST and
The Victoria University of Manchester
                                                                    Outline

          Metcalfe’s Law, Reed’s Law and Group Forming Networks


          Collaboration Grids as GFNs for Virtual Organisations


          Examples from provider and user communities


          Conclusions




Combining the strengths of UMIST and
The Victoria University of Manchester
                                        21st March 2007   Reading             2
                                                                        Notions of Grid
          Collaboration Grids
                 – Multiple institutions, secure, widely distributed, VOs
                 – Service level agreements & commercial partnerships




                                                                                  Increasing Complexity and Revenue
                 – Business model: Increase overall revenue
               Enterprise Grids
                  – Virtualization of enterprise resources and applications
                  – Aggregation and centralization of management
                  – Business model: Reduce total cost of ownership
               Clusters
                  – Networks of Workstations, Blades, etc.
                  – Cycle scavenging, Homogeneous workload
                  – Business model: Lower marginal costs
               Parallel Processing Systems
                  – Parallel processing for single applications

               Slide courtesy of David Snelling, Fujitsu
Combining the strengths of UMIST and
The Victoria University of Manchester
                                        21st March 2007               Reading                                         3
                                                                                Metcalfe’s Law

          Metcalfe’s law:
                  – the value of a telecommunications network is proportional to the
                    square of the number of users
                  – n2 – n ~ n 2
                            • the number of pair-wise connections enabled by the network


          Benefit to a user grows as the number of other users: n – 1
          Collective value is the number of users (n) multiplied by the benefit
           each receives


          Footnote:
                  – Odlyzko and Tilly (2005) suggested that not all connections are
                    equally valuable to a user, so the value of the network should actually
                    grow as n log n

Combining the strengths of UMIST and
The Victoria University of Manchester
                                          21st March 2007                     Reading            4
                                          Reed’s Law and Group Forming
                                                             Networks
         David P. Reed, 1999
                            • Context Magazine article ―Weapon of Math Destruction‖
                            • later, ―That Sneaky Exponential — Beyond Metcalfe's Law to the Power of
                                 Community Building‖


         Reed suggested that the value of some networks grows with
           the number of (non-trivial) sub-groups that can be formed
           from the network participants. This is:
                            2n – n – 1 ~ 2n

         He called these Group Forming Networks, and cited e-Bay
           as an example.

Combining the strengths of UMIST and
The Victoria University of Manchester
                                           21st March 2007              Reading                         5
                                                     Implications of Reed’s Law




                                                            Source: D. P. Reed,
                                                          That Sneaky Exponential


         Value of Network Type                     Gain in value by merging two networks
         n                                         (n+m) – (n+m) = 0
         n2                                        (n+m)2 – (n2 + m2) = 2nm
         2n                                        2(n+m) – (2n +2m)
                                                   If n=m, 2n(2n-2) ~ (2n)2
Combining the strengths of UMIST and
The Victoria University of Manchester
                                        21st March 2007                             Reading   6
                                                                                                   Caveat!




                   Not all connections are equally valuable to a user, so the value of a network grows more slowly.
                   Perhaps Metcalfe’s law should be n log n, not n2.
Combining the strengths of UMIST and
The Victoria University of Manchester
                                        21st March 2007                       Reading                                 7
                                        Sharing and Virtual Organisations
       Share (v):
             To have part; to receive a portion; to partake, enjoy, or suffer with others.
       Need more enjoyment, less suffering.



                              R         R         ?                       R
                                                                                                  VO C
                                                                   R
                                                                                              R   R        ?
                                        R                                     R
                     VO A


                                                                                                  R       R
                                                                          R
                                                                                    ?
                  R                                                                                   R
                                        ?         R           R
                 R                                                        VO B
                            R

                                                                  "…enables communities (“virtual organizations”) to share
            “flexible, secure, coordinated resource               geographically distributed resources as they pursue
            sharing among dynamic collections of                  common goals -- assuming the absence of central
            individuals, institutions, and resources"             location, central control, omniscience, existing trust
                                                                  relationships."
Combining the strengths of UMIST and
The Victoria University of Manchester
                                            21st March 2007                    Reading                                8
                                                                     Grids as GFNs for VOs

        InterGrid                         Virtual                Science       Science                  Collaboratories
                                         Research                  2.0         Gateway
                                        Environment                               s
                                             s                                             Communit
                                                                      Reed’s               y Building
                      Virtual                                          Law
                   Organisations,
                    VOMs, CAS




                  Are Collaboration Grids the Group Forming Networks
                                 for Virtual Organisations?


Combining the strengths of UMIST and
The Victoria University of Manchester
                                               21st March 2007                   Reading                                  9
                                                                                       Not so simple
          Grids and VOs are complex, multi-dimensional networks
                  – people, networks (of the other kind), resources (machines, data,
                    applications, services,...)
                            • I’m being deliberately vague when I talk about the size of such networks
          Glued together by social networks
          Most examples of effective multi-organisational VOs today
                  – build on established social networks and have long term goals and
                    strategy
                  – have significant funding, governance & policy of their own
          These VOs
                  – need to share data, applications, instruments,...
                  – sometimes collectively control some (computational) resources
                  – often have access rights to resources they do not control
                            • e.g. national HPC services, public data sets
                  – may or may not have the capacity to construct their own grids

Combining the strengths of UMIST and
The Victoria University of Manchester
                                           21st March 2007                       Reading                 10
                                                               What is the domain of facile VO
                                                                                   formation?
         Consider
          production grids (TeraGrid, EGEE, UK NGS, DEISA,...)




                                                                                                                                                 Increasing Size and Value
                  –     single sign-on and interoperable middleware
                            •    tend to have policy, governance, monitoring/accounting/logging, centralised (or federated) support, training,
                                 documentation, access/account lifecycle procedures...
                            •    things generally not addressed by middleware!
                  –     (almost) sufficient, but limited in size and value
               ―islands of interoperability‖
                  –     larger domains of single sign-on and interoperable middleware can arise without
                        central governance
                  –     but interoperation not sustainable without concerted effort
                  –     no-coordinated policy, governance, monitoring/accounting/logging, centralised
                        support, training, documentation, access/account lifecycle procedures...
                  –     cost of joining is higher for users; problem resolution is hard
               ―Intergrid‖ or ―Grid of grids‖
                  –     doesn’t exist yet


               Internet
                  –     necessary infrastructure, but not sufficient


Combining the strengths of UMIST and
The Victoria University of Manchester
                                                   21st March 2007                                       Reading                                                             11
                                          RealityGrid – an example of a VO
               A UK e-Science Pilot Project (2002-2005), continuing through various funding
               Common goals:
                  – use of simulation software and other applications in condensed matter physics and
                    molecular modelling
                  – themes of computational steering, visualization, coupled models
                            •    development and use of methods, tools, & middleware that support these
               Relies mostly on others for resource provision
                  – NGS (including CSAR, HPCx, HECTOR), TeraGrid, DEISA, departmental facilities,
                    UKLight
                  – Needs co-ordinated use of resources across these
                            •    advanced reservation, co-allocation,...
               Has some services of its own
                  – originally for computational steering, parameter space exploration, job migration
                  – now also for virtualised applications (e.g. Application Hosting Environment)
                            •    to improve scalability of client deployment and insulate from middleware incompatibilities between grids
                            •    relying on JSDL and GridSAM (an OGSA-BES prototype) for this


Combining the strengths of UMIST and
The Victoria University of Manchester
                                                21st March 2007                             Reading                                         12
                                                                                  TeraGyroid
             Funding from NSF (US) and EPSRC (UK) for projects linking TeraGrid and NGS
             Winner HPC Challenge (2003) and ISC Award (2004)

                                                          TeraGyroid at SC Global
                                                                            2003




                                                                                  TRICEPS

                                                                                  Winner SC’03
                                                                                  HPC Challenge


Combining the strengths of UMIST and
The Victoria University of Manchester
                                        21st March 2007                 Reading                   13
                                                                    SPICE
    transport of biomolecules across protein
     membrane channels
    For a 300,000 atom system, takes ~30M
     CPU-hours to simulate 10 microseconds
     using MD techniques
    SPICE reduces this by 50-100 by using
     a technique called SMD-JE
             – introduces two new parameters which
               must be tuned
                      • using many shorter simulations
                      • these can be run in parallel
             – tuning facilitated by computational
               steering
             – uses a RealityGrid-modified version of
               NAMD
    Awards: HPC Analytics SC’05, ISC 2006
Combining the strengths of UMIST and
The Victoria University of Manchester
                                        21st March 2007   Reading       14
                                            Interoperation is a global
                                                                 issue
                              AIST-GTRC
    CSI (JP)
                                         NAREGI (JP)                               EGEE(EU)


                                                                GridPP(UK)
                                                                                      LCG(EU)

             DEISA(EU)                        common users,
IBM                                           staff & procedures             GGF          Condor(US)
                                                                                                       OSG(US)
                                          common                                   Globus(US)
         UniGrids(EU)                     users        NGS(UK)

                                                                    common         TeraGrid(US)
                                                   common           users
                                                   funding
                                        OMII(UK)
                                                                             NMI(US)
Combining the strengths of UMIST and
The Victoria University of Manchester
                                              21st March 2007                   Reading                     15
                                                             Early lessons from GIN


                                                                                                       GIN
         As Mark Linesch’s grandfather used to say
                ―GIN, GIN, GIN, it makes me want to sin[g]‖

               Grid Interoperation Now! is a GGF Community Group
               authentication and authorisation is on the critical path
                  –     GIN has >10 participating grids
                  –     experimenters from each grid need access to resources in all grids
               AuthN: Need to recognise certificates from many Certification Authorities
                  –     International Grid Trust Federation (IGTF), a federation of regional Policy Management
                        Authorities (PMA), covers around 100 CAs
                  –     If we didn’t have it, we’d need to invent it
               AuthZ: VOMS from EGEE tremendously useful for streamlining authorization
                  –     each grid determines how it will support the GIN VO
                  –     hugely improves scalability
               Issues:
                  –     already encountering VO name clashes (DNS-like solution proposed)
                  –     need Acceptable Use Policies, contacts etc for the VO


Combining the strengths of UMIST and
The Victoria University of Manchester
                                         21st March 2007                       Reading                           16
                                                          NGS Partnership Programme

         Goals:


         1.         Increase the range and depth of services and resources that NGS
                    can offer to its users


         2.         Provide leadership and guidance to sites needing to put their
                    resources ―on the Grid‖




Combining the strengths of UMIST and
The Victoria University of Manchester
                                        21st March 2007         Reading               17
                                                                                    Partners and Affiliates

         Resource providers join NGS as Partners or Affiliates
          Partners provide significant resources or services to NGS users
                  – collect usage statistics (accounting)
                  – have formal Service Level Descriptions and production-quality support
                    arrangements
                  – have representation in NGS governance
               Affiliates retain total control over who has access to their resources
               Both
                  – require approval by NGS Board (meets quarterly)
                            •    on basis of site’s SLD, ―buddy’s‖ report, and Technical Board recommendation
                  – are subject to continued monitoring and review
                            •    partnership and affiliate status can be withdrawn by NGS Board
                  – exchange support and security contacts with NGS helpdesk
                            •    users need a single point of contact for UK Grid
                  – in order to ensure that the NGS brand connotes reliable, production services to
                    users



Combining the strengths of UMIST and
The Victoria University of Manchester
                                               21st March 2007                            Reading               18
                                                                               NGS Compliance
         Principle:
          If many NGS sites offer the same kind of service, they should use compatible interfaces
             and protocols

               These are prescribed in the NGS Software Stack
                  –     try to avoid mandating particular software versions, and verify compliance through test suites
               Partners and affiliates are certified for compliance
                  –     approval requires passing compliance test suite for seven days
                  –     continuous monitoring and regular review


         ―One-off‖ and experimental services are tested, but not covered by NGS Software Stack. All
            are vetted for:
          consistency with NGS security policies, and
          complementarity with NGS core services.




Combining the strengths of UMIST and
The Victoria University of Manchester
                                          21st March 2007                       Reading                                  19
                                                                   Partners with specialist offerings


        Westminster operates and supports P-GRADE portal and GEMLCA services for NGS
          users, in addition to the usual Globus services




         Belfast e-Science Centre (approved December 2006) provides:
               Basic Execution Services
                  –     GridSAM instances configured for all NGS resources
                            •    OMII-UK job submission software

               Service Hosting Service
                  –     BeSC manage various web service containers into which projects or VOs can deploy their
                        own Web or Grid services



Combining the strengths of UMIST and
The Victoria University of Manchester
                                                 21st March 2007               Reading                           20
                                                          NGS & Partners, 2006




Combining the strengths of UMIST and
The Victoria University of Manchester
                                        21st March 2007         Reading          21
                                                            Membership pipeline
         (not a complete list)
               Partners
                  – GridPP sites, initially Imperial, Glasgow
                  – Condor/Windows at Cardiff
               Affiliates
                  – NW-Grid/Manchester SGI Prism
                  – Microsoft HPC Cluster at Southampton
                  – HECToR
                  – SunGrid
               Data partners (early discussions)
                  – EDINA and MIMAS
               Others in discussion




Combining the strengths of UMIST and
The Victoria University of Manchester
                                        21st March 2007         Reading       22
                                                                    Why join?

          Institutions have a mission to support their own users


          Increasing dependence on computation & data, and growing need to
           collaborate beyond the institution, make ―getting on the Grid‖ a
           necessity for an institution to remain competitive


          NGS shows the way.




Combining the strengths of UMIST and
The Victoria University of Manchester
                                        21st March 2007   Reading             23
                                                          What’s in it for a partner?

               They get NO resources (people or hardware) from NGS (except support)
               They have to expend effort to integrate their resources and support
                arrangements into NGS
               They derive some value from NGS brand, expertise, support networks, hand-
                holding,...
               They might hope to ―trade‖ un-utilized cycles for other resources on the ―NGS
                market‖, but this is not possible yet
               Joining should be approximately cost-neutral for an affiliate.

               But partners as asked to donate resources to the pool, and they have no
                control over their disposition
                  – altruistic at the best of times,
                  – and now we have fEC!


         Are we pinning the future of the NGS to hopes of an epidemic of altruism?


Combining the strengths of UMIST and
The Victoria University of Manchester
                                        21st March 2007        Reading                          24
                                                             What’s in it for a partner, again?

          Need to restore to partners some measure of control over the
           disposition of resources in the NGS pool.
          Currently working towards a model in which:
                  – NGS creates a VO (using VOMS) for each partner site
                  – and associates with it a project allocation
                  – the partner has full control over the membership of the VO
                  – any member of the VO can access NGS partner resources (affiliates
                    could elect out of bands to support this VO)
                  – the allocation is either:
                            • primed in proportion to the donated resources, and/or
                            • topped up in proportion to usage of the partners’ resources made by other sites
          Implied need for accounting, audit, ―exchange rates‖,...



Combining the strengths of UMIST and
The Victoria University of Manchester
                                           21st March 2007                      Reading                         25
                                                           Virtual Organisations in NGS
               Many different understandings of VO

               In the NGS view, there is
                  –     a consumer-provider relationship between the VO and the Grid
                  –     end users are members of (one or more) VOs


               VOs bring value to their members by:
                  –     sharing applications / tools / data
                  –     perhaps providing a community-oriented view of the Grid
                  –     negotiating community access rights with providers
                  –     ...
               NGS is moving steadily towards VO-based approaches to:
                  –     access management (need VO notion to make processes scale)
                            •    project-based requests for resources, accounting and authorisation
                                        –   resource requests from individual users will continue to be accepted
                            •    resource providers decide what VOs to ―support‖
                  –     and user engagement
                            •    more effective to help communities with common needs than individuals



Combining the strengths of UMIST and
The Victoria University of Manchester
                                                    21st March 2007                                   Reading      26
                                                                                        SHEBANGS
               Shibboleth-Enabled Bridge to Access the NGS
               Leverage widespread Shibboleth infrastructure at institutions
               Map user’s Shibboleth attributes onto short-lived VOMS credentials that a portal presents to the grid




Combining the strengths of UMIST and
The Victoria University of Manchester
                                        21st March 2007                      Reading                                    27
                                                                     Conclusions and Claims
               Grids are not yet very effective at supporting VO formation.
               Reed’s Law suggests that the long-term value proposition for collaboration grids depends on how well
                they facilitate this
                  –     and provides compelling arguments for combining or federating grids
               Interoperation is a pre-requisite for federation
                  –     standards work and conscious co-operation are essential
               Gaps in middleware & standards are patched by policy & practice within a grid
                  –     When federating grids, problems re-surface.
               Already, the VO notion is proving useful to user and provider communities alike as these scale up
                  –     despite there being almost no tools focused on VO support
               VOs need to become first-class entities in grids
                  –     Make explicit the consumer-provider relationship between VOs and grids
                  –     VOs need to have global scope – their value is severely diminished when constrained to a single grid
               NGS Partnership programme important for expansion of UK Grid
                  –     and integration of specialist resources and services (e.g. datasets, visualization, instruments)
               NGS as integration fabric
                  –     could evolve into unifying mechanism for UK compute provision?
               Need to take small steps towards market economics?
                  –     Incentivise partners
                  –     Accommodate fEC


Combining the strengths of UMIST and
The Victoria University of Manchester
                                               21st March 2007                             Reading                             28
                                                                    Last word


                Collaboration Grids MUST become the Group Forming
                            Networks for Virtual Organisations




                                                                    GIN

Combining the strengths of UMIST and
The Victoria University of Manchester
                                        21st March 2007   Reading           29
                Questions?




Combining the strengths of UMIST and
The Victoria University of Manchester
                                                           OGF20 7-11 May 2007

          Manchester, UK
                  – Manchester Central


          Co-located with 2nd EGEE User Forum


          Hosted by UK e-Science
                  – local support from the University of Manchester




Combining the strengths of UMIST and
The Victoria University of Manchester
                                        21st March 2007        Reading       31

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:1
posted:8/9/2011
language:English
pages:31