Docstoc

HPC-infrastructure GRID-initiatives

Document Sample
HPC-infrastructure GRID-initiatives Powered By Docstoc
					               GRID-initiatives in the Nordic region
Conference xxx - August 2003

                  status and future challenges
                                       Anders Ynnerman
                                     EGEE Nordic Federation
                       Director Swedish National Infrastructure for Computing
                                       Linköping University
                                             Sweden
        Outline of presentation

• Status in the Nordic countries
    • Denmark, Finland, Norway, Sweden
•   SweGrid testbed for production
•   NorduGrid
•   Nordic DataGrid Facility
•   Nordic Grid Consortium
•   North European Grid
•   Identifying potential problems for Nordic Grid
    collaborations




                            Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 2
                  DENMARK
           HPC-infrastructure                                      GRID-initiatives

•   HPC program initiated 2001                           •   Copenhagen University is a partner
     •   Danish Center for Scientific Computing              in NorduGrid
         (DCSC)
     •   Distributed program                             •   Danish Center for Grid Computing
•   Location of facilities based on                          (DCGC) has just been established
    scientific evaluation of participating                     •   Virtual center located a 5 sites
    groups not on computer centers or                          •   Main location at the Niels Bohr Institute,
    infrastructure                                                 Copenhagen University
     •   Århus                                                 •   0.75 MEuros/year
     •   Odense                                                •   Operations and support of a Danish Grid
     •   DTU                                                   •   Research agenda with 10 senior
     •   Copenhagen University                                     researchers and 5 PhD students
•   Budget                                                     •   Strong ties to NorduGrid and Nordic
     •   2.1 MEuros (2001)                                         DataGrid Faicility
     •   1.6 MEuros (2002)                               •   Participation in EGEE through NBI
     •   1.5 MEuros (2003)
     •   1.4 MEuros (2004)




                                                  Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 3
                 NORWAY
          HPC-infrastructure                          GRID-initiatives

•   NOTUR – The Norwegian High-              •   Metacenter goal – To establish a
    Performance Computational                    uniform and as seamless as
    Infrastructure                               possible, access to HPC resources
     •   Trondheim                           •   Parallab at Bergen is participating in
     •   Oslo                                    the Nordic Grid Consortium with
     •   Bergen
                                                 PDC and CSC
     •   Tromsö
                                             •   Parallab is contributing to EGEE
•   Budget
                                                 security team
     •   168 MEuros (2000-2003)
     •   50% from reserach councils          •   Bergen and Oslo are partners in
     •   Statoil, DNMI                           NorduGrid
     •   In kind contributions               •   Plans for a ”NORGRID” are being
•   New program similar to the Swedish           developed
    program is being initated and placed
    under the NREN (UNINETT)


                                      Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 4
                  FINLAND
          HPC-infrastructure                                       GRID-initiatives

•   Center for Scientific Computing                       •   CSC is one of the founding partners
    (CSC) is the (main) center in                             in the Nordic Grid Consortium
    Finland
                                                          •   CSC is contributing resources to
•   The largest HPC center in                                 NorduGrid
    Scandinavia
                                                          •   CSC is a partner in DEISA
•   Runs services for academic users
                                                          •   Helsinki Institute of Physics is
    all over Finland and FMI
                                                              running several GRID projects and
•   Budget                                                    is contributing to the EGEE security
     •   Directly from the ministry of education              team
                                                          •   A Finnish Grid dedicated to
•   Smaller university HPC-systems                            Condensed Matter Physics is being
    (clusters) are beginning to appear                        built




                                                   Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 5
                 SWEDEN
          HPC-infrastructure                                   GRID-initiatives

•   Swedish National Infrastucutre for                •   Lund and Uppsala are partners in
                                                          NorduGrid
    Computing (SNIC) formed during
    2003                                              •   Largest resource contributions to
                                                          NorduGrid are SNIC clusters at
•   6 participating centers                               Umeå and Linköping
     •   Umeå
                                                      •   SweGrid production Grid is being
     •   Uppsala                                          built (600 CPUs) over 6 sites 120 Tb
     •   Stockholm                                        storage
     •   Linköping
                                                      •   SNIC is hosting the EGEE Nordic
     •   Göteborg
                                                          Regional Operations Center
     •   Lund
                                                      •   Stockholm (PDC) is coordinating
•   National resource allocation (SNAC)                   the EGEE security activity
•   Budget                                            •   PDC is one of the founders of the
     •   4.5 MEuros/year (Government)                     Nordic Grid Consortium
     •   4.0 MEuros/year private foundations          •   PDC is a partner in the European
                                                          Grid Support Center


                                               Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 6
SweGrid
                 •   3 parts
                       •   Research and development
                       •   Operations
                       •   Hardware
                 •   Total budget 3.6 MEuro
                 •   6 GRID nodes:
                 •   600 CPUs
                       •   IA-32, 1 processor/server
                       •   875P with 800 MHz FSB and dual
                           memory busses
                       •   2.8 GHz Intel P4
                       •   2 Gbyte
                       •   Gigabit Ethernet
                 •   12 TByte temporary storage
                       •   FibreChannel for bandwidth
                       •   14 x 146 GByte 10000 rpm
                 •   1 Gigabit direct connection to
                     SUNET (10 Gigabit)



          Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 7
         Persistent storage on SweGrid?




     1                  2                                       3


Size                                                    Bandwidth
Administration                                          Availability



                     Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 8
                The NorduGrid project

• Initiated by several Nordic universities
    •   Copenhagen, Lund, Stockholm, Oslo, Bergen, Helsinki, later: Linköping and Umeå
• Started in January 2001 & funded by NorduNet-2
    •   Simultaneously with EU DataGrid, but with only a 2 years budget and 3 new positions
    •   Initial goal: to deploy DataGrid middleware to run “ATLAS Data Challenge”
• Cooperation with DataGrid
    •   Common Certification Authority and Virtual Organization tools, Globus2 configuration
    •   Common applications (high-energy physics research)
• Switched from deployment to R&D in February 2002
    •   Deployed a light-weight and yet reliable and robust Grid solution in time for the
        ATLAS DC tests in May 2002
• Continuation
    •   NorduGrid was one of the driving forces behind the creation of the ”North European
        Grid Federation” together with the Dutch Grid, Belgium and Estonia
    •   Is an option for middleware for the ”Nordic Data Grid Facility”
    •   …as well as for the Swedish Grid facility SWEGRID



                                           Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 9
The resources
                • Currently available resources:
                     •   4 dedicated test clusters (3-4 CPUs)
                     •   Some junkyard-class second-hand
                         clusters (4 to 80 CPUs)
                     •   Few university production-class
                         facilities (20 to 60 CPUs)
                     •   Two world-class clusters in Sweden,
                         listed in Top500 (230 and 440
                         CPUs)
                • Other resources come and go
                     •   Canada, Japan – test set-ups
                     •   CERN, Russia – clients
                     •   It’s open, anybody can join or part
                • People:
                     •   the “core” team grew to 7 persons
                     •   local sysadmins are only called up
                         when users need an upgrade




           Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 10
A NorduGrid snapshot




           Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 11
                 Reflections on NorduGrid
•   Bottom up project driven by an application motivated group of talented people

•   Middleware adaptation and development has followed a flexible and minimally
    invasive approach

•   HPC centers are currently “connecting” large resources since it is good PR for
    the centers

•   As soon as NorduGrid usage of these resources increases they will be
    disconnected. There is no such thing as free cycles!

•   Motivation of resource allocations is missing
     •   NorduGrid lacks an approved procedure for resource allocation to VOs and individual user groups
         based on scientific reviews of proposals
•   Funding for the continuation of the project is today unclear
     •   NorduGrid may remain a middleware project that delivers solutions to other Nordic production
         Grids such as SweGrid




                                              Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 12
             Nordic DataGrid Facility
• Based on a Working paper by the NOS-N working group
   (April 4, 2002): road map towards the Nordic Grid Facility
• Project started 2003-01-01
• Builds on interest from
   •   Biomedical sciences
   •   Earth sicences
   •   Space and astro sciences
   •   High energy physics
• Gradual build-up
   • Project leader and 4 postdocs
   • Projects in middleware and application interfaces




                                   Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 13
              Nordic DataGrid Facility
                                International Evaluation
                                   Recommendations
                                Competition for location
                                 of Nordic Grid Facility


                                                        Nordic Agreement
           Competence Building                   Investments in Nordic Grid Facility
      Nordic Funding for co-ordination
           of Nordic co-operation                        Capacity Building
                                                           Consolidation




      2002        2003             2004            2005            2006           2007

Prototype period 2002-2004:                       Capacity building 2004-2007:
Scope 0.54 MEuro.                                 Scope 15 MEuro.



                                           Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 14
          NDGF challenges


  Users    Users    Users     Users
 Country 4Country 3Country 2 Country 1            Current HPC setup



                                                                 Funding agency
                                                                    country 1
                                                             Funding agency
       HPC-center
HPC-center             HPC-center
               HPC-center                                       country 2
                                                          Funding agency
                                                             country
                                                     Funding agency 3
                                                         country 4




                               Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 15
              VO 3
                                                   NDGF?
                                    Proposals

             Users
      VO 1
                                                                              Other GRIDs
                     VO 2
                                                GRID management


                Accounting       Allocations
Middleware
                                                                        MoU

                             SLAs
        HPC-center
         HPC-center                                   Funding agency
          HPC-center                                   Funding agency
           HPC-center                                    Country 1
                                                        Funding agency
                                                          Country 1
                                                          Funding agency
                                                            Country 1
                                                             Country 1


                               Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 16
             Nordic Grid Consortium
• PDC, CSC and Parallab have signed Memorandum of
  Understanding.
• The Vision is to:
   • “create a Grid for the benefit of the users of the facilities”


• Purpose is to:
   • Explore Grid production from an HPC-center point of view
   • Explore means of exchanging resource allocations across national borders


• NGC is an inclusive organisation and will expand to include
  all major HPC centers in Scandinavia



                                      Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 17
              Conclusions
• A large number of GRID initiatives in the Nordic Region
• HPC – centers and GRID initatives are rapidly merging
• Strong need for a mechanism for exchange of resources
  over the borders
• Very few existing users belong to VOs
   • Most cycles at HPC-centers are used for chemistry
   • How will these users become GRID users?
• There is a need for authorities that grant resources to
  projects (VOs)
   •   Locally
   •   Regionally
   •   Nationally
   •   Internationally


                                 Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 18
            North European Grid
• NEG is an EGEE federation consisting of
   • Belgium, Denmark, Estonia, Finland, Holland, Norway, Sweden
• NEG is responsible for the security in EGEE
• NEG is hosting an EGEE Regional Operations Center
   • Two lead sites
       SARA
       SNIC/PDC

• NEG executive committee has been formed
   • One representative per partner in the federation plus also a national
     representative per country
   • Phone meetings every second Friday to deal with common issues
• NEG will play an important role in the coordination of Grid
  efforts in the federation

                                   Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 19
            The NorduGrid philosophy
• No single point of failure
• Resource owners have full control of the contributed
  resources
• Installation details should not be dictated
   • Method, OS version, configuration, etc.
• As little restriction on site configuration as possible
   • Computing nodes should not be required to be on the public network
   • Clusters need not be dedicated
• NorduGrid software should be able to use existing system
  and Globus installation
   • Patched Globus RPMs are provided
• Start with something simple that works and proceed from
  there


                                  Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 21
 Nordic
 Atlas                     MoU     NorduGrid
  User
 Group

                           MoU     SweGrid
 Nordic




                  NDGF
                           MoU     DCGC
PigMap
Members
                           MoU     Notur

                           MoU     CSC

  Dalton
                                       ...
  Users



          Atlas               PigMap


                  Workshop on eInfrastructure (Internet and Grids) Rome Dec-03 - 22

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:2
posted:11/24/2012
language:English
pages:21