Gabriel

Document Sample
Gabriel Powered By Docstoc
					Problem Solving Environments
and Application Development in
           the Grid

                         Edgar Gabriel
           High Performance Computing Center Stuttgart
                        gabriel@hlrs.de
GFK 2 Workshop, Seoul                           Edgar Gabriel
                        Höchstleistungsrechenzentrum Stuttgart
Outline

• What is the Grid
• Problem solving environments in Grids
   – Cactus
   – COVISE
• ASP in Grids
   – UNICORE
• Software development in the Grid
   – The DAMIEN project
   – Programming techniques
   – A case study: PCM
   – The GrADS project
• Summary

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Thanks to


For providing (at least partially) the slides

    –    Grid-computing: Michael Resch
    –    COVISE: Uwe Wössner
    –    UNICORE: Stefan Wesner and Ralf Ratering
    –    GrADS: Graham Fagg




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
                            What is the Grid ?




GFK 2 Workshop, Seoul                           Edgar Gabriel
                        Höchstleistungsrechenzentrum Stuttgart
Computer, Networks, Experts




                                 National Legion Net
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Computational grid vs. electric power grid




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
The Grid

•    Distributed Computing
       – synchronous processing
•    High-Throughput Computing
       – asynchronous processing
•    On-Demand Computing
       – dynamic resources
•    Data-Intensive Computing
       – databases
•    Collaborative Computing
       – science
Ian Foster and Carl Kesselman, editors, “The Grid: Blueprint for a New Computing
Infrastructure,” Morgan Kaufmann, 1999

    GFK 2 Workshop, Seoul                           Edgar Gabriel
                            Höchstleistungsrechenzentrum Stuttgart
Definitions
•    The metacomputer is a network of heterogeneous, computational
     resources linked by software in such a way that they can be used
     as easily as a personal computer.
     (L. Smarr and C.E. Catlett, „Metacomputing”, Communications of
     the ACM 35/6, 45-52, 1992.)

•    A computational grid is a hardware and software infrastructure
     that provides dependable, consistent, pervasive, and inexpensive
     access to high-end computational capabilities.
     (Ian Foster & Carl Kesselman, „The Grid“, MK, 1998.)

•    “Grid Problem”, which we define as flexible, secure, coordinated
     resource sharing among dynamic collections of individuals,
     institutions, and resources.
     (Ian Foster, Carl Kesselman and Steven Tuecke, “The Anatomy of
     the Grid”,Int. Journal of Supercomputing Applications, 2001.)


    GFK 2 Workshop, Seoul                           Edgar Gabriel
                            Höchstleistungsrechenzentrum Stuttgart
Goals

• “as easy to use as a PC”

•    reliable
•    dynamic
•    secure
•    cheap
•    multi-institutional




    GFK 2 Workshop, Seoul                           Edgar Gabriel
                            Höchstleistungsrechenzentrum Stuttgart
Technologies for the management of resources

• Definition of resource types:
   – Computing
   – Data
   – Network
   – Instruments
• High level services:
   – Resource discovery
   – Resource scheduling
   – Resource brokering
• Monitoring
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Networks

• (Still) current bottleneck of Grids
• Gbit-Networks in place
   – USA: vBNS, Abilene, ESNet
   – Germany: G-WIN (2.5 GBits)
   – France, GB, Italy, Netherlends: similar
     activities
   – Interconnecting national networks:
        • STAR TAP
        • Géant
• Technology: moving away from ATM
   – Quality of Service ?
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Géant

•Developed from TEN-155
•Fully operational in November 2001
•Covering 31 countries
•Interconnecting over 3,000 institutions




                                                                  Cyprus




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
                                                                  Israel
A small part of the ongoing projects...
• Europe                                                     • Germany
   – DataGrid                                                    – UNICORE+
   – EUROGRID                                                • Italy
   – DAMIEN                                                      – http://sara.unile.it/grb/
   – FP6
                                                             • France
• USA
                                                                 – Action Concertée
   – GLOBUS
                                                                   Incitative GRID (ACI
   – Legion                                                        GRID)
   – IPG
                                                             • Great Britain
   – GrADS
                                                                 – UK-HEP Grid
• Japan
   – Ninf                                                        – e-science
   – TME

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Problem Solving Environments in
            the Grid




GFK 2 Workshop, Seoul                           Edgar Gabriel
                        Höchstleistungsrechenzentrum Stuttgart
What is a Problem Solving Environment

• Something that helps you to solve a problem
• An integrated environment supporting the entire
  life cycle to solve a problem with easy access by
  an end-user
• Solving of a problem consists of
    – development
    – execution
    – analysis




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
More detailed...

• Development
   – Tools to help problem specification, design, analysis
     and verification
   – Rapid protoyping
   – Dependence on a specific domain
   – Expert assistence
• Execution
   – Support online/offline observation
   – in the Grid: activities performed on multiple
     heterogeneous components
       • selection, testing, configuration, activation,
         monitoring
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Overview about some existing PSEs
  Name           Application                        Base technology   Web-site
                 domain

  Cactus         Astrohysics                                      -   www.cactuscode.org




  TENT           Aerospace                          CORBA/Java        www.dlr.de/tent




  JACO3          Aerospace                          CORBA/Java        www.irisa.fr/paris/nfrancais/jac
                                                                      o3.htm



  COVISE         Visualization                                    -   www.hlrs.de/organization/vis/c
                                                                      ovise



  SCIRun         Medical simulation                               -   software.sci.utah.edu/scirun.ht
                                                                      ml




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
                                         CACTUS

             slides based on the Cactus tutorial
           available at http://www.cactuscode.org/

GFK 2 Workshop, Seoul                           Edgar Gabriel
                        Höchstleistungsrechenzentrum Stuttgart
• CACTUS is a PSE for enabling large scale
  distributed computing in the Grid
• Original application area: numerical relativity,
  astrophysics
• Development by AEI Potsdam, ZIB, NCSA and
  many more
• Open source (GPL)




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Architecture of Cactus

            Cactus
                                                                  Thorns
            Computational
                                                  Toolkit         Toolkit
              Toolkit
                                                Flesh

                  Configure                     Make              CST

          SuperUX                                             Irix  Linux
                UnicosOperating                         Systems HP-UX
           Solaris  AIX                                       OSF   NT

                                                                   Picture source: Cactus tutorial
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
The Flesh offers


• Parameter parser with types, ranges, checking
• Scheduling of routines
• Make system on many architectures
• Dynamic argument lists
• Utilities such as program checking test suites




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
A thorn is

• a piece of code offering a certain functionality, which you can
  use in your application (or is part of your application)
• Consists of
   – source code
   – parameter files
   – configuration files
   – make information
   – documentation
   – testing
• Thorns can be grouped in arrangements for organizational
  reasons (e.g. IO related routines)
• A collection of arrangements is called a toolkit (e.g. the
  computational toolkit)
  GFK 2 Workshop, Seoul                           Edgar Gabriel
                          Höchstleistungsrechenzentrum Stuttgart
Application view

  Application                                                     Computational
   Toolkit                                                          Toolkit




                                        Application
                                                                           Computational
                                         Toolkit
                                                                             Toolkit




                    CCTK_(…)                            Flesh               CST
Picture source: Cactus tutorial
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Configuration files for a thorn

• CCL: Cactus Configuration Language
    – object oriented
• interface.ccl: describes the interface of the thorn to the
  flesh
• param.ccl: description of the parameters of this thorn
• schedule.ccl: describes when this thorn shall be
  executed (e.g. whether another has to be called before
  this thorn can be executed)

• The code which will be compiled is generated by a perl-
  script in the Cactus Specification Tool (CST)

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
In your code

• you should use Cactus Datatypes
         e.g. CCTK_REAL, CCTK_INT, ...
• you should use the Cactus wrappers for
  communication (support for several communication
  methods)
         e.g. CCTK_Reduce, CCTK_Barrier
• argument lists
  subroutine WaveToyF77_Evolution(CCTK_ARGUMENTS)
  DECLARE_CCTK_ARGUMENTS

• to create a new thorn: gmake <newthorn> (will
  create all necessary subdirectories etc.)
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
The Computational Toolkit

• Contains all arrangements and thorns which you
  (probably) need for a regular scientific application
   – Driver functions (storage, communication)
   – IO
   – Coordinates

• Precise list of the available functionality and
  parameters: ThornGuide




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
How to get Cactus

• Web-page www.cactuscode.org contains extensive
  documentation
• Download script: GetCactus
• Takes a Thornlist as argument
• Several examples on the web-page, eg:
   – ThornListAll.th: all available thorns
   – ComputationalToolkit.th : the thorns of the
     Computational Toolkit
• Structure of the Thornlist (arrangement/thorn):
  CactusBase/CartGrid3D
  CactusBase/Boundary
  CactusBase/Time
  CactusPUGH/PUGHSlab
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
How to install and use Cactus

• Specify a thornlist to be compiled
• Further options, e.g. used MPI library, compiler
  settings, etc.
• configure and make executed automatically
• subdirectory with the name of the configuration
  created (e.g. configs/myconfig)
• Example:
gmake myconfig -THORNLIST=mythornlist
  -MPI=MPICH -MPICH_ARCH=LINUX
  -MPICH_DIR=/usr/local/mpi -MPICH_DEVICE=ch_shmem



 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Running an executable

• At successful compilation, an executable
  exe/myconfig will be created

• A parameter-file is needed on startup
   – which thorn shall be activated
   – set values for parameters




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
 Output example (I)                                                 Source: Cactus tutorial
> mpirun -np 2 exe/cactus_wave wavetoy.par
-----------------------------------------------------------------------------
       10
  1   0101       ************************
  01 1010 10       The Cactus Code V4.0
 1010 1101 011      www.cactuscode.org
  1001 100101    ************************
    00010101
     100011     (c) Copyright The Authors
      0100      GNU Licensed. No Warranty
      0101

-----------------------------------------------------------------------------
Activating thorn Cactus...Success -> active implementation Cactus
Activating thorn iobasic...Success -> active implementation IOBasic
Activating thorn idscalarwave...Success -> active implementation idscalarwave
Activating thorn time...Success -> active implementation time
Activating thorn wavetoyf77...Success -> active implementation wavetoy
Activating thorn pugh...Success -> active implementation driver
Activating thorn pughreduce …Success -> active implementation pughreduce
Activating thorn cartgrid3d...Success -> active implementation grid
Activating thorn ioutil...Success -> active implementation IO
Activating thorn ioascii...Success -> active implementation IOASCII
   GFK 2 Workshop, Seoul                           Edgar Gabriel
                           Höchstleistungsrechenzentrum Stuttgart
Output example (II)                                                Source: Cactus tutorial

Startup routines
    CartGrid3D: Register GH Extension for GridSymmetry
    CartGrid3D: Register coordinates for the Cartesian grid
    PUGH: Startup routine
    PUGHReduce: Startup routine
    IOUtil: Startup routine
    IOASCII: Startup routine
    IOBasic: Startup routine
    WaveToyF77: Register banner

  Parameter checking routines
    CartGrid3D: Check coordinates for CartGrid3D
    IDScalarWave: Check parameters

  Initialisation
    CartGrid3D: Set up spatial 3D Cartesian coordinates on the GH
    IOASCII: Choose 1D output lines
    IOASCII: Choose 2D output planes
    PUGH: Report on PUGH set up
    Time: Set timestep based on Courant condition
    WaveToyF77: Schedule symmetries
    IDScalarWave: Initial data for 3D wave equation
  GFK 2 Workshop, Seoul                           Edgar Gabriel
                          Höchstleistungsrechenzentrum Stuttgart
 Output example (III)                                    Source: Cactus tutorial
do loop over timesteps
    WaveToyF77: Evolution of 3D wave equation
    WaveToyF77: Boundaries of 3D wave equation
    t = t+dt
    if (analysis)
    endif
  enddo
  Termination routines
    PUGH: Termination routine
  Shutdown routines
------------------------------------------------------------------------
Driver provided by PUGH
------------------------------------------------------------------------
WaveToyF77: Evolutions of a Scalar Field
------------------------------------------------------------------------
INFO (CartGrid3D): dx=>3.0000000e-01 dy=>3.0000000e-01 dz=>3.0000000e-01
INFO (CartGrid3D): x=>[-0.150,8.550] y=>[-0.150, 8.550] z=>[-
   0.150,8.550]
INFO (PUGH): MPI Evolution on 4 processors
INFO (PUGH): 3-dimensional grid functions
INFO (PUGH):       Size: 30 30 30
INFO (PUGH):       Processor topology: 1 x 2 x 2
INFO (PUGH):       Local load: 7680     [30 x 16 x 16]
INFO (PUGH): Seoul
   GFK 2 Workshop,                Edgar 0.000000
                   Maximum load skew: Gabriel
                Höchstleistungsrechenzentrum Stuttgart
Output example (IV)                                                Source: Cactus tutorial

--------------------------------------------
 it |          | phi                       |
     |    t    | Min          Max          |
--------------------------------------------
   0 |    0.000| 0.00000000 | 0.99142726 |
  10 |    1.500| -0.14944313 | 0.00005206 |
  20 |    3.000| -1.16992203 | 0.00000000 |
  30 |    4.500| -0.72347868 | -0.00000002 |
  40 |    6.000| -0.41127922 | -0.00000094 |
  50 |    7.500| -0.29834817 | -0.00002427 |
  60 |    9.000| -0.23577373 | -0.00035044 |
  70 |   10.500| -0.19532474 | 0.00000311 |
  80 |   12.000| -0.16688911 | 0.00010812 |
  90 |   13.500| -0.14582895 | 0.00022987 |
 100 |   15.000| -0.12957000 | 0.00033802 |
 110 |   16.500| -0.11631732 | 0.00025752 |
 120 |   18.000| -0.06945640 | 0.00019346 |
---------------------------------------------------------
Done.



  GFK 2 Workshop, Seoul                           Edgar Gabriel
                          Höchstleistungsrechenzentrum Stuttgart
More features of Cactus

• Available thorns implement needed important
  functionality (e.g. IO routines, different solvers
  etc...)
• Support for many packages (e.g. Globus, PAPI,
  PETsc, etc...)

• remote monitoring
• checkpoint-restart




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
More information about Cactus

• Web site: www.cactuscode.org
• Extensive documentation available:
   – Users Guide
   – Thorn Guide
• Complete tutorial on the webpage
• Many publications and presentations available




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
                                           COVISE
COllaborative VIsualization and Simulation
               Environment


                              slides by Uwe Wössner
                                       HLRS
GFK 2 Workshop, Seoul                           Edgar Gabriel
                        Höchstleistungsrechenzentrum Stuttgart
COVISE overview
  COVISE developed since 1993
  Virtual Reality developed since 1997

   Major Aims:
    - Access to remote supercomputing resources
    - Remote consulting, collaborative working (CSCW)
    - Modular and extensible architecture
    - Post-processing for major application areas (CFD)
    - Handling of large data volumes
    - Direct 3D interaction from within VEs
   Features:
     - Distributed object oriented data management
     - Visual application builder
     - Collaborative user interface
     - Scalable from desktop to IPT
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
   • founded in October 1997
   • by nine employees of the Computing Center
     at the University of Stuttgart
     (ranging from Student up to Professor)


What do we offer?
   • COVISE environment for the analysis of numerical
   simulations
   • and the experience and know-how of the founders



GFK 2 Workshop, Seoul                           Edgar Gabriel
                        Höchstleistungsrechenzentrum Stuttgart
COVISE, COllaborative VIsualization and Simulation
Environment




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
COVISE Architecture

 Remote Workstation                            Local Workstation           Remote Supercomputer

           UI                                            UI



                                                    Controller



RENDER                                      RENDER                FILTER          SIM



            2‘                                     2              1‘               1



          CRB                                          CRB                        CRB




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
COVISE Modules

             Hosts

     Categories
     Modules




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Simple Pipeline


                                           Scalar Data
Structured Grid




Surface Polygons                             Scalar Data on Surface




                                            Colors on Surface nodes




                                     Polygons with Colors, no Normals




     GFK 2 Workshop, Seoul                           Edgar Gabriel
                             Höchstleistungsrechenzentrum Stuttgart
Parameters




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Color Modules




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Isosurface




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
VectorField




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
IO Modules

    GenDat                                  GenerateTest Data
    RWCovise                                Read and write binary COVISE data
    ReadAbaqus                              ABAQUS
    ReadDyna3D                              Crash
    ReadFireUniversal                       Fire ASCII format
    ReadFluent                              Fluent UNS / Fluent 5.x
    ReadKiva                                Kiva Combustion Simulation
    ReadMovieBYU                            Crash
    ReadNastran                             Nastran
    ReadPatran                              Patran Neutral File
    ReadStar                                Star CD all Versions
    ReadTascflow                            Tascflow
    ReadTecplot                             Tecplot general
    ReadVectis                              Vectis V6 (Stationary)
    ReadVectisTri                           Vectis Surfaces (ASCII)
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
More Modules

 Filter
     CuttingSurface                          Plane or Cylindrical Cut
     CuttingPlane                            Cut in index space in structured grids
     FilterCrop                              Reduce the size of structured grids
     MinMax                                  Calculate min an d max values for Colormaps
     SimplifySurface                         Reduce number of polygons
     SmoothSurface                           Smooth polygonal surfaces


 Mapper
     IsoLines                                Iso lines on polygonal surfaces
     Isosurface                              Isosurfaces on structured and unstructured grids
     VectorField                             Show vector arrows



 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
More Modules

 Tracer
   CellTracer                                Tracer for vectis data
   MagTracer                                 Tracer for unstructured grids
   TetraTrace                                Tracer for tetrahedra grids

 Color
   Colors                                    Color Map (standard or textured)
   ColorEdit                                 User defined color map



  Renderer
    COVER                                     Virtual Environment Renderer
    Renderer                                  Inventor Renderer
    VRMLRenderer                              Generates VRML 2.0 files


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Tools

    CellToVert                              Convert cell centered data to vertex data
    Check USG                               Remove duplicate vertices from unstructured grids
    Collect                                 Collect Geometry, Color and Normal objects
    CutGeometry                             Cut polygonal surfaces
    Domainsurface                           Calculate bounding surface of an unstructured grid
    GenNormals                              Generate smooth normals for polygonal surfaces
    GridSurface                             Convert 2D structured grid to polygons
    ShowGrid                                Show structured grids or elements of USGs
    Sphere                                  Draw spheres around points
    Tetrahedronize                          Create tetrahedra grid from unstructured grids
    Transform                               Move, rotate and mirror polygonal surfaces and USGs
    Tube                                    Draw tubes around lines
    VectorScal                              Extract scalar data from vertex data




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Interaction with COVISE modules




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Libraries for Normal Application Modules


                                       Application Code
                                     COVISE Api 2.0
                                     (libcovise_api)

COVISE Application
Library(libappl)

                   COVISE Basic Library (libcovise)



 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
The COVISE Execution Model




                                                                  •   data flow oriented

                                                                  • connected modules form
                                                                    a pipeline

                                                                  • data and information
                                                                    flow from top to bottom



 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Module Execution Scheme


                    Constructor
                               Connection to COVISE



                         PostInst


                                             Parameter change
                                                                   param()
                         Main                  Execution event
                                                                  compute()
                         Loop
                                                 Socket data
                                                                  sockData()


                           Quit



 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
The Setup Phase



    Describe                     Describe                        Setup
     ports                                                    connections     PostInst
                                 parameters




         Input Port Optional Input Port               controller      datamanager




       Ouput Ports



 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
The Compute Method




                     read input                   Do                  Write
                     param/data                computation         output data




                  communication                                   communication
                       with                                            with
                   datamanager                                     datamanager




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
COVISE Data Types


   • Structured/Unstructured Grids

   • Structured/Unstructured Scalar Data

   • Structured/Unstructured Vector Data

   • Colors/Normals

   • Points/Lines/Tristrips/Polygons

   • Sets

   • User Defined.......


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
COVISE Parameter Types and Interactors

  • Boolean

  • String

  • Scalar

  • Vector

  • Choice

  • Slider

  • Browser




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Integration of Simulation Codes
         Black Box                                Grey Box            White Box




 characteristics:                        characteristics:         characteristics:
 • no source code                        • linkable code          • source code available

 examples:                               examples:                  examples:
 • batch tools                           • commercial packages    • FENFLOSS/FLOW (IHS)
                                           (STAR-CD, FIDAP,...)   • URANUS

 typical coupling:                       typical coupling:        typical coupling :
 • file interface                        • SimLib,                • as COVISE Module
 • control via scripts                     „user subroutines“     • SimLib

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Simulation Library

                  COVISE Session                                  Simulation

         Coupling module
                                                                       Simulation code
                     Start
                     Exec
                     Para
                     Data                                              User subroutines


         Simulation Server                                             Simulation Client

          COVISE system                                                  System calls
                                                       TCP
               Network                              connection             Network




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
COVER COvise Virtual Environment Renderer

Main Features:
   – All common tracking devices
   – All possible screen configurations
   – Collaborative working
   – VRML97               covise_lib render_lib                                 libJS

                           ModuleDesk




                              Dynamil               ModuleSupport
                                                                    COVER       VRML97




                             Viewpoint


                                                        VRUI        VRBClient   Performer




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Collaboration modes
•Loose
    -Viewpoints are not synchronized
    -Avatars show the position of partners
    -Suited for joint work
•Tight
    -Viewpoints are synchronized
    -Avatars are not displayed by default
    -All Partners are allowed to interact with the VR World
    -Suited for joint work on small objects and presentations
•Master/Slave
    -Viewpoints are synchronized
    -Only the master is allowed to interact with the VR world
    -Suited for Presentations


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Grid-Computing Scenario

         PSC                                       HLRS                      MSC
                            PACX/MPI                              PACX/MPI




     Wall at SC2000                                                     CAVE at HLRS




                                              COVISE
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Reentry of Space Vehicle (X38)




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
World-wide Collaborative VR




•    Sandia National Labs, Albuquerque                               •   UNI•C, Copenhagen
•    Centre For Virtual Environments, Salford                        •   VR-Medialab, Copenhagen
•    Cerfacs, Toulouse                                               •   PDC, Stockholm
                                                                     •   NCHC, Taiwan
•    CSAR, Manchester                                                •   Fraunhofer IAO Stuttgart
•    UCL, London


    GFK 2 Workshop, Seoul                           Edgar Gabriel
                            Höchstleistungsrechenzentrum Stuttgart
GFK 2 Workshop, Seoul                           Edgar Gabriel
                        Höchstleistungsrechenzentrum Stuttgart
UNICORE outline




•    Concept
•    Submission of a simple job
•    UNICORE-plugins
•    Installation of UNICORE




    GFK 2 Workshop, Seoul                           Edgar Gabriel
                            Höchstleistungsrechenzentrum Stuttgart
UNICORE

• Uniform Interface to COmputing Resources
• Goals:
   – Seamless, secure and intuitive access to HPC
     resources
   – Consistent batch access to different remote
     systems
   – Support existing and emerging technologies
   – Minimal intrusion into the centers
   – Support multi-system and multi-site applications



 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
UNICORE status

• Current version: 3.6
• Version 4.0 available at the end of the year
• Licensing:
   – Protocoll will be publicly available
   – Distribution and support by Pallas GmbH

• Free download of testsystem at:




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Partner

             Forschungszentrum                                    Rechenzentrum der
             Jülich                                               Universität Stuttgart

             Pallas GmbH                                          Leibniz-
                                                                  Rechenzentrum
             Deutscher                                            München
             Wetterdienst
                                                                  Universität-GH
                                                                  Paderborn
             University of
             Karlsruhe
                                                                  Konrad-Zuse-
                                                                  Zentrum Berlin
             Technical University of
             Dresden
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Architecture Overview

 JPA (Job Preparation Agent)                                      Other client (e.g. NIFTI)

                                           SSL

                                         Gateway                   UNICORE Server



          Network Job Supervisor
                                                            Other servers (e.g. SAFE)
                  (NJS)



      Batch subsystem via TSI                                Target system access
                                                             mechanism (e.g. SSH)


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
UNICORE Acronyms

• Usite: a site providing UNICORE services (e.g.
  HLRS)
• Vsite: computing resources (e.g. a single computer,
  an SMP or a cluster)

• Uspace: dedicated file space on the target system;
  may only exist during the execution of the job
• Xspace: permanent storage on the execution system
  (e.g. home, file-server)
• Nspace: file system on the local computer, where the
  client is running
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Job Submission

JPA (Job Preparation Agent)                                       HTTP www.unicore.de
            SSL
                                                         Trusted        Is user Cert signed by trusted CA
                Gateway                                    CAs

  Socket/SSL
                                                                   UUDB       Map CERT to Unix Login
    Network Job Supervisor
            (NJS)
         Socket                                                                 Make generic job
                                                                  IDB
                                                                                machine specific

                               TSI
                                                                          Job Storage Space
            Supercomputer                               USpace


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Multi-site Jobs
          UNICORE GUI

                [Plug-in]
                                                    HTTP www.unicore.de
                           SSL
         UNICORE Server                                               UNICORE Server
                                                        SSL
               Gateway                                                    Gateway

   Network Job Supervisor                                          Network Job Supervisor
                               TCP/IP                                            TCP/IP
         Batchsystem
        Batchsystem                                                       Batchsystem
                                                                         Batchsystem
 Batch subsystem via TSI                                          Batch subsystem via TSI

           UNICORE Site 1                                              UNICORE Site n

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
JPA/JMC screenshot




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
A script job - script editor




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
A script job - options




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
A script job - file imports and exports




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Job Monitoring




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Technology Details (I)

• Client is implemented in Java
• The protocol is implemented in two layers
   – Unicore Protocol Layer (UPL)
     Simple Request/Response Protocol realized
     with serialized Java Objects
   – Abstract Job Objects (AJO) model Jobs such
     as a Script Task in a generic way are
     transmitted as serialized Objects within UPL
     Requests



 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Technology Details (II)




                                                                  Deserialize
                           Serialize
 Script Task                           10010010001111110...                     Script Task
   Signature                                       SSL                           Signature


UPL Request                                       TCP/IP                    UPL Request




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
UNICORE File Transfer (FTM)

                   NIFTI
               SSL
                                                          Trusted     Is user Cert signed by trusted CA
                   Gateway                                  CAs
     Socket/SSL
                                                                    UUDB     Map CERT to Unix Login
    Secure Access File Executive
              (SAFE)


RSH/SSH with suexec

                                               XSpace


                                                  Storage Server

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Technology Details (FTM)


• Architecturally integrated with the rest of the
  UNICORE software
• Protocol is also based on serialized Java Objects
• Due to different nature of file transfer commands
  the protocol is a multi stage protocol (not UPL)
• Will Use the same Gateway




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
UNICORE plug-in mechanism

• Application-specific support in UNICORE
• Currently planned/supported plug-in:
    • CPMD
    • Fluent
    • STAR-CD
    • Nastran
    • Gaussian
    • ...




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Graphical Plugin Integration (1)




                                                                  Source: Ralf Ratering, Pallas GmbH
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Graphical Plugin Integration (2)



 Menu Items




                                                                  Source: Ralf Ratering, Pallas GmbH
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Graphical Plugin Integration (3)



 Menu Items

 Representation
  in Job Editor




                                                                  Source: Ralf Ratering, Pallas GmbH
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Graphical Plugin Integration (4)



 Menu Items

 Representatio in
  Job Editor

 GUI




                                                                   Source: Ralf Ratering, Pallas GmbH
  GFK 2 Workshop, Seoul                           Edgar Gabriel
                          Höchstleistungsrechenzentrum Stuttgart
    Graphical Plugin Integration (5)


    Menu Items

    Representation
     in Job Editor

 GUI
    Representation
     in Job Monitor


                                                                         Source: Ralf Ratering, Pallas GmbH
        GFK 2 Workshop, Seoul                           Edgar Gabriel
                                Höchstleistungsrechenzentrum Stuttgart
  Handling of plugins for users (1)


User
                                                                •   Download plugin jar
                         Client                                     archive and store in
                       Software                                     plugin directory.

                       Plugin XY
         Plugin
       Jar Archive

            Website                                                 Source: Ralf Ratering, Pallas GmbH
   GFK 2 Workshop, Seoul                           Edgar Gabriel
                           Höchstleistungsrechenzentrum Stuttgart
  Handling of plugins for users (2)

                                                                    Source: Ralf Ratering, Pallas GmbH
User
             Client                                     2.Ist the plugin author trustworthy?
           Software
           Plugin XY




   GFK 2 Workshop, Seoul                           Edgar Gabriel
                           Höchstleistungsrechenzentrum Stuttgart
  Handling of plugins for users (3)
                                                                      Source: Ralf Ratering, Pallas GmbH
User
                                                                3.Editing the plugin
                         Client                                   parameters.
                       Software
                       Plugin XY




   GFK 2 Workshop, Seoul                           Edgar Gabriel
                           Höchstleistungsrechenzentrum Stuttgart
  Handling of plugins for users (4)
                                                                     Source: Ralf Ratering, Pallas GmbH
User
                                                                    4.Will the selected
                             Client                                   virtual site support
                           Software                                   the plugin?

                           Plugin XY

       Request                             Software Resource:
                                           Plugin XY

                 Virtual Site
   GFK 2 Workshop, Seoul                           Edgar Gabriel
                           Höchstleistungsrechenzentrum Stuttgart
  Handling of plugins for users (5)
                                                                     Source: Ralf Ratering, Pallas GmbH
User
                                                                    5.Submit the job.
                         Client
                       Software
                       Plugin XY
                                           Submit Job containing
                                           Plugin tasks

                            Virtual
   GFK 2 Workshop, Seoul                           Edgar Gabriel
                           Höchstleistungsrechenzentrum Stuttgart
Plugin development


• 3 classes to be implemented:
   – TaskContainer: Hides AJO action graph from user
   – JPAPanel: Graphical interface for parameters
   – IUnicorePlugin: Start&stop plugin

• UNICORE Client Plugin Programmer„s Guide
describes the classes which have to be implemented
very good


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
UNICORE Containers

                                             ActionContainer



  JobContainer                        TaskContainer                  ServiceContainer                 ...



         FileContainer                                  ExecuteContainer



Import, Export ...                                                UserContainer



                           PluginContainer                            Source: Ralf Ratering, Pallas GmbH


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Signing Plugin Jars (1)

     Plugin Writer                                                scriptPlugin.jar
                                                                    Ralf Certificate
                                                                                     Certificate Chain
   Ralf Certificate                                                 CA Certificate
                                         jarsigner
   Ralf Private Key
   CA Certificate

     Client Keystore



        Client User

   Thomas Certificate
   Thomas Private Key
   CA Certificate
                                                                       Source: Ralf Ratering, Pallas GmbH

      Client Keystore
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Signing Plugin Jars (2)
        Plugin Writer                                                  Source: Ralf Ratering, Pallas GmbH


     Ralf Certificate
     Ralf Private Key
     CA Certificate

       Client Keystore



          Client User

     Thomas Certificate                                           scriptPlugin.jar
     Thomas Private Key
     CA Certificate
                                                       ?            Ralf Certificate
                                                                    CA Certificate

        Client Keystore
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Signing Plugin Jars (3)
     Plugin Writer
                                                                       Source: Ralf Ratering, Pallas GmbH

   Ralf Certificate
   Ralf Private Key
   CA Certificate

     Client Keystore



        Client User

   Thomas Certificate                                         scriptPlugin.jar
   Thomas Private Key                                             Ralf Certificate
   CA Certificate
                                                    ?             CA Certificate

      Client Keystore
                                                       Yes
 GFK 2 Workshop, Seoul                           Edgar Gabriel    Accept Ralf Certificate?
                         Höchstleistungsrechenzentrum Stuttgart
Signing Plugin Jars (4)
     Plugin Writer                                                     Source: Ralf Ratering, Pallas GmbH


   Ralf Certificate
   Ralf Private Key
   CA Certificate

     Client Keystore



        Client User

  Thomas Certificate
                                                              scriptPlugin.jar
  Thomas Private Key
  CA Certificate                                                  Ralf Certificate
  Ralf Certificate                                                CA Certificate

      Client Keystore
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
UNICORE Installation

• Acquire UNICORE certificates
• Components:
   – client
   – gateway
   – UUDB (UNICORE User Data Base)
   – NJS (Network Job Supervisor)
   – TSI (Target System Interface)
• Requirements:
   – java 1.3
   – perl 5
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
UNICORE certificates

• Required certificates:
   – User certificate (client)
   – Server certificates (NJS, Gateway)
• UNICORE-CA at LRZ Munich
   – https://unicore-ca.lrz-muenchen.de
   – Netscape 4 currently required
• Apply for User-certificate for each user
• Apply for Gateway and NJS certificates for new
  gateway or NJS
• Valid for one year

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Installing the Client

• For UNIX:
   – package delivered as a tar-file
   – unpack the file
   – edit in the bin-directory the unicore-script and
     change the UNICORE-PATH

• For Windows
   – package delivered with an installer
     unicore_<version>_setup.exe


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
At first start

• Keystore for saving certificates has to be
  generated
• export your User-certificate from Netscape
  (password required!)
• download UNICORE-CA certificate from the Web
• import User and UNICORE-CA certificate in the
  client
• check in the preferences directory the user
  settings (e.g. add your e-mail address)



 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
UNICORE Certificates




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Installing the Gateway (I)


• Machine on it‟s own, which is not used for anything
  else
• Example install-path: /usr/local/unicore
       cd /usr/local/unicore
       tar -xvf ~/Unicore/Gateway_3.0.tar
       cd Gateway_3.0




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Installing the Gateway (II)

• Adapt files:
   – bin/run_gate
  UNICORE_GW=${UNICORE_GW:-usr/local/unicore/Gateway3_0}
  PATH=/usr/j2se/bin:$PATH
  JAVA=/usr/j2se/bin/java config/
    – gateway.properties
    gw.server_certs=<link to the server certificate>
    gw.trusted_cas=<link to the downloaded CA>
    gw.password=....
    – config/vsites
    Hitachi dimrill.VsiteC ... njs.uni-stuttgart.de 8181


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Installing the UUDB
   cd /usr/local/unicore/
   tar -xvf ~/UNICORE_3.5/UUDB_3.5.tar
   cd UUDB_3.5
   export PATH=/usr/j2se/bin:$PATH
   export JAVA =/usr/j2se/bin/java
• First time installation
   ./installer ....
• Update
   ./installer -u /usr/local/unicore/UUDB /usr/local/unicore/NJS_3.5.1/lib
• Add a User to the UUDB:
   ./bin/uu_add ${HOME}/unicore-edgar.crt ruseg




  GFK 2 Workshop, Seoul                           Edgar Gabriel
                          Höchstleistungsrechenzentrum Stuttgart
Installing the NJS

• you might use the same certificate like for the
  Gateway
  cd /usr/local/unicore
  tar -xvf ~/Unicore/NJS_3.0.tar
  cd NJS_3.0
  mkdir certs
  mv ~/njs1.p12 certs/
  cp /usr/local/unicore/Gateway_3.0/ certs/unicore-ca-cert.crt certs
• Edit config/njs.properties
• Edit config/IDB/HitachiIDB



 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Installing the TSI

• Has to be installed on the target machine
• Adaptations for most platforms and batch-systems
  available
• Copy the appropriate tsi-file into the bin directory
  on the target system
• Set permissions correctly
  mkdir -p /usr/local/unicore/bin /usr/local/unicore/bin/tsi
  chmod 755 /usr/local/unicore
  chmod 750 /usr/local/unicore/bin
  chmod 1777 /usr/local/unicore/bin/tsi
  file hi-ux_tsi ==> /usr/local/unicore/bin


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Starting UNICORE

• Start the Gateway
  cd /usr/local/unicore/Gateway_3.5.1/config
  nohup ../bin/run_gate &
• Start the NJS
  cd /usr/local/unicore/NJS_3.0.12/config
  nohup ../bin/run_njs &
• Start the TSI on the target system
  cd /usr/local/unicore/TSI_3.5.1
  nohup ./bin/run_tsi /usr/local/unicore/TSI_3.5.1/TSI/hitachi/tsi
           gw.rus.uni-stuttgart.de 4711 &




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
More information

• UNICORE Web-site:
       http://www.fz-juelich.de/unicore
• UNICORE-Forum:
       http://www.unicore.org
• Contact: info@pallas.com
• Support: unicore-support@pallas.com
• Documentation:
   – UNICORE Client description (56 pages)
   – UNICORE Client Plugins - Programmers Guide
     Version 3.6 (13 pages)
   – Installation guides to all components available
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Future of UNICORE

• Eurogrid
   – European UNICORE based testbed for three
     application areas
• GRIP
   – interoperability between UNICORE and Globus

• Convergence towards OGSA (Open Grid Software
  Architecture)




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
     The DAMIEN Grid Software Development
     Environment: Standards, Tools, Libraries


GFK 2 Workshop, Seoul                           Edgar Gabriel
                        Höchstleistungsrechenzentrum Stuttgart
Motivation

• Industrial and academical applications have high
  requirements for memory and CPU‟s
• Typical problems: multi-physics coupled
  applications (e.g. fluid-structure interaction)
• Large companies have sites and computing
  resources all over the world




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Problems


• How to write applications for Computational-Grids ?
• Users of Parallel Computers are used to
   – standards (e.g. MPI)
   – libraries
   – tools
• These are (partially) not yet available for Grid-
  environments



 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Goals


• provide a toolbox starting from existing and widely
  accepted tools to support Grid-environments
• implement the necessary changes
• test the toolbox with real applications from industry
  in a testbed




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
DAMIEN partners




                          CRIHAN / Rouen                                  Pallas GmbH / Brühl

                                                                            HLRS / Stuttgart
                            EADS CCR / Paris




                                                                  CEPBA / Barcelona




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Software life-cycle for a Grid-application

• Two major stages
   – Development phase: modify/optimize an
     application for Grid-computing
   – Production phase: use the stable application for
     production runs

• An application usually passes both stages several
  times




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
The development phase
                     • Sequential code(s)
                     • Parallel (MPI) code(s)
                                                                  Starting point

                     • parallelization
                     • code coupling
                     • optimisation

                     • compiling
                     • linking with libraries
                                                                     Code
                     • debugging
                                                                  development
                     • performance analysis

                     • testing


                                         Results           no
                                          ok ?
                                             yes
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
The development phase
                     • Sequential code(s)
                     • Parallel (MPI) code(s)

                     • parallelization                                  MPI
                     • code coupling
                     • optimisation                                   MpCCI

                     • compiling                                      MpCCI
                     • linking with libraries
                                                                    PACX-MPI

                     • debugging                                    MetaVampir
                     • performance analysis
                                                                  MetaVampirtrace

                     • testing


                                         Results           no
                                          ok ?
                                             yes
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Programming Models for Grids
 Model                            Example                         Implementation

 Data Parallel                    MPI                             PACX-MPI, MPICH-G2,
                                                                  Stampi
 Task Parallel                    GridRPC                         Ninf, Netsolve, Nimrod

 Distributed                      CORBA,                          ...
 Objects                          JavaRMI
 Component                        ...                             CCA, Meta-Chaos
 Framework
 Application                                                      Cactus, P3T
 Framework


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
MPI and GRIDS

• Good:
    –    Low level communication primitives
    –    MPI Datatypes support different data representations
    –    MPI communicators allow flexible grouping of processes
    –    Collective Operations allow to abstract different
         communication patterns

• What is missing ?
  – Static (MPI-2)
  – Information about the used infrastructure (MPI2-JoD)
  – Fault tolerance (FT-MPI)
  – Quality-of-Service handling (MPI-RT, MPICH-GQ, PACX-
    MPI)

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
MpCCI: Coupling Model




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
MpCCI: Basic Functionality

• Communication
   – Based on MPI
   – Coupling of Sequential and Parallel Codes
   – Communicators for Codes (internal) and Coupling (external)

• Neighbourhood Search
   – Bucket Search Algorithm
   – Interface for User-defined Neighbourhood Search

• Interpolation
   – Linear Surface Interpolation for Standard Elements
   – Volume Interpolation
   – User-defined Interpolation
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
PACX-MPI

 0       1         2           3          4           5
                                                                   Internal MPI-communication
                                                                   on MPP 1 using Vendor-MPI

      Out-daemon                     In-daemon


                                                                   External communication
      In-daemon                      Out-daemon                    (e.g. TCP/IP, ATM, SSL)

                                                                   Internal MPI-communication
  6        7        8            9            10          11
                                                                   on MPP 2 using Vendor-MPI

         Node of MPI_COMM_WORLD
 GFK 2 Workshop, Seoul                             Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Point-to-point operation in PACX-MPI

          MPP-1                                                            MPP-2

     2            0              Out-                               In-     5      7
                                daemon                            daemon

                                  In-                              Out-
     3            1                                                         4      6
                                daemon                            daemon


           header
           data
Point-to-point operation between global node 2 and global node 4


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Status of the Implementation (I)

• Full MPI 1.2 implemented
• MPI-2 functionality
   – Extended collective operations
   – Language interoperability routines
   – Canonical Pack/Unpack Functions
• MPI 2 JoD Cluster attributes
• Other implemented features:
   – data conversion
   – data compression
   – data encryption
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Status of the implementation (II)

• PACX-MPI ported and tested on
    –    Cray T3E
    –    SGI Origin/Onyx
    –    Hitachi SR2201 and SR8000
    –    NEC SX4 and SX5
    –    IBM RS6000/SP
    –    SUN platforms
    –    Alpha platforms
    –    LINUX-PC„s



 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
MetaVampir - Application Level Analysis




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
MetaVampir - Application Level Analysis

                 MPI Application
                                             PACX-MPI calls
                MetaVampirtrace                                    single Tracefile
                                             PACX-MPI profiling interface calls
                      PACX-MPI
MPI calls
   Native MPI          Network
                      protocols
• MetaVampir will show application information
• Application MPI processes
• Application communication

  GFK 2 Workshop, Seoul                           Edgar Gabriel
                          Höchstleistungsrechenzentrum Stuttgart
MetaVampir - Middle–Layer Analysis

                         MPI Application
                                     PACX-MPI calls
                                                   PACX-MPI internal
                       PACX-MPI
                                                communication routines
                                     MPI calls
                     MetaVampirtrace                Tracefile

                                                        PMPI calls
  QoS–Layer        Native MPI
• Additional information
   – PACX-MPI daemons                                             - QoS information
   – Internal communication                                       - MpCCI information
   – Copy, en/decode overhead


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
The production phase
                                                        Experience with      Results of deve-
Given problem         Grid-enabled code                small problem sizes
                                                                              lopment phase


           Determine optimal number of processors
                and combination of machines
                                                                             Production
                                                                                 run
                                Execute job




  GFK 2 Workshop, Seoul                           Edgar Gabriel
                          Höchstleistungsrechenzentrum Stuttgart
The production phase
                                                        Experience with      MetaVampirtrace
Given problem         Grid-enabled code                small problem sizes
                                                                              Dimemastrace

                                                                                Dimemas
           Determine optimal number of processors
                and combination of machines                                    MetaVampir



                                                                              Configuration
                                Execute job                                     Manager

                                                                              QoS Manager




  GFK 2 Workshop, Seoul                           Edgar Gabriel
                          Höchstleistungsrechenzentrum Stuttgart
Dimemas

• Application performance analysis and prediction
  tool for message passing programs
• In development since 1992
• On a workstation
• Dimemas currently distributed
  by CEPBA




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Dimemas: Tuning Methodology

                                     Sequential                    Tracing     MP library
                                      machine                                  - MPI                Message
                                                                                                    Passing
                                                                   Tracing     - PVM                 Code
                                                                  facilities   - etc...
                                 Parallel machine
  Dimemas
    Trace
                                                                                            Code modification
     File




      DIMEMAS                                                                                Paraver
                                                           Visualization
                                                              Trace
                                                               File



                                                                                            Visualization
      Simulation                                                                            and analysis


                                                             Parameters
                                                             modification
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Configuration Manager

• Specification of the Grid-computer
• Create the PACX-MPI
  configuration files
• Distributed the
  configuration-files
• Start and monitoring
  the application




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Configuration Manager –GridTerminal
Execute command on remote hosts




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Configuration Manager – File Transfer
• Generic Interface
• Supported protocols
   – FTP
   – SCP
   – FTM




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
QoS Manager                                                       CRIHAN

• Objectives
  – Provide a dynamic adaptation of network
    resources (QoS) for a distributed MPI-
    application
  – Reservation of network resources
      • Quality of Service over Native ATM
      • Quality of Service over IP (IPv4 or IPv6)
  – Specification of the QoS manager functionality
      • Provide a practical QoS allocation for
        applications
      • Help end-users with the selection of QoS

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
DAMIEN tools in the production phase
                                                       Dimemas-Tracefile

                                                     Edit new configuration

                                                 Execute Dimemas simulator


Check results with Vampir                             All configurations
                                                           tested ?           no

                                                                  yes
                                                Specify best configuration

                                                    Specify QoS parameter

                                                       Launch job with
                                                    Configuration Manager
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
DAMIEN End-User application

• EADS distributed over numerous sites all over
  Europe
• Computing resources are distributed  “natural”
  need for Grid-software to couple the resources
• Coupled vibro-acoustic simulations
   – structure of
   rockets during the
   launch
   – noise reduction
   in airplanes-cabins

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
  Programming techniques for Grid
          environments




GFK 2 Workshop, Seoul                           Edgar Gabriel
                        Höchstleistungsrechenzentrum Stuttgart
•    Building local communicators
•    Avoiding small messages
•    Overlapping communication and computation
•    Variable handling of ghost-cells

• The GrADs project: Scalapack on the Grid




    GFK 2 Workshop, Seoul                           Edgar Gabriel
                            Höchstleistungsrechenzentrum Stuttgart
Building communicators per machine
• Goal: set up a communicator which contains all processes on
  one machine
• Useful for
   – load balancing
   – coupled applications
• PACX-MPI: MPI2-JoD Cluster attributes
   MPI_Attr_get (MPI_COMM_WORLD, MPI_CLUSTER_COLOR,
                 &color);
   MPI_Comm_split (MPI_COMM_WORLD, color, key,
                 &newcomm);

• IMPI: similar approach, but three instead of two levels
• MPICH-G2: similar approach, but not restricted to a certain
  number of levels
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Collection of small messages (I)

 Linear algorithm for collecting data from all processes
 on a single root (MPI_Gather): every process sends its
 data explicitly

Host 1                     0               3                      6   9    Host 2

                           1               4                      7   10


                           2               5                      8   11




 Example: 2 hosts, 6 processes each
 • 6 messages between the two hosts
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Collection of small messages (II)

 Host-based algorithm for collecting data from all processes :
 collect all data for a host and send it as a single message


Host 1                     0               3                      6   9    Host 2

                           1               4                      7   10


                           2               5                      8   11




 Example: 2 hosts, 6 processes each
 • 1 message between the two hosts
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Collection of small messages - performance




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Switching protocol

• IMPI
   – calculate the sum over all messages using on
     the root node and broadcast the protocol
   – if (sum < limit) use host-based algorithm
     else use linear algorithm
• PACX-MPI approach
   – determine on each machine the local sum and
     use to this machine the required protocol
   – not using to all hosts the same protocol!


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Switching protocol - performance (I)
       MPI_Gatherv using 3 hosts with 32-32-16 nodes
       PACX-MPI: no global broadcast required




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Switching protocoll - performance (II)

         MPI_Gatherv using 3 hosts with 32-32-16 nodes




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Overlapping communication and computation
• E.g. Matrix-Vektor Multiplikation on a regular grid using 1-
  ghost cell
   – values of the neighbor cells required




 Problem
 • Values from the neighbor required


  GFK 2 Workshop, Seoul                           Edgar Gabriel
                          Höchstleistungsrechenzentrum Stuttgart
Overlapping communication and computation

• E.g. Matrix-Vektor Multiplikation on a regular grid using 1-
  ghost cell




 Usual Algorithm
 • Data exchange
 • Execute the Matrix-Vector Multiplication at once

  GFK 2 Workshop, Seoul                           Edgar Gabriel
                          Höchstleistungsrechenzentrum Stuttgart
Overlapping communication and computation (II)
Alternative algorithm:
• Initiate communication (MPI_Isend/MPI_Irecv)
 • Calculate inner values
 • Finish communication (MPI_Waitall)
 • Calculate boundary values




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Overlapping computation and communication (III)

• Example: performance of an MPI-parallelized
  equation solver on a Cray T3E
      No of                Bi-CGSTAB             Bi-CGSTAB- TFQMR         TFQMR-split
      processes            [sec]                 split [sec] [sec]        [sec]

      1                    16.54                 16.92            44.07   45.36

      2                    9.57                  10.83            28.88   32.74

      4                    4.90                  5.48             13.23   15.10

      8                    2.52                  2.82             6.78    7.75

      12                   2.06                  2.58             5.66    7.30

      16                   1.72                  2.21             4.73    6.22

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Overlapping computation and communication (IV)

• The same equation solvers on two Cray T3E
  coupled by PACX-MPI
       No of     Bi-                              Bi-             TFQMR   TFQMR-
       processes CGSTAB                           CGSTAB-         [sec]   split [sec]
                 [sec]                            split [sec]
       2                    17.33                 13.95           50.36   39.07

       4                    9.05                  6.60            25.24   17.22

       8                    5.87                  4.39            15.85   11.23

       12                   6.24                  4.22            17.73   10.71

       16                   6.03                  4.68            16.87   12.06
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Scale-up behavior




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Using more than one ghostcell

• Work done by the Cactus Group (SC 2001)
• Idea: use several n-ghostzones instead of one and
  communicate only every n Iteration
• Example for n =2
   First iteration                            Second iteration    Communication




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Summary for programming techniques

• No general rules which can be followed
• Dynamic adaptation of the algorithms required
• Try to define high-level abstract interfaces for
  communication relevant function
• Use collective operations when feasible - and
  check whether it is really faster

• Two more important issues covered in the case
  study:
   – load-balancing
   – communication pattern of the application

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
                                 A Case study




GFK 2 Workshop, Seoul                           Edgar Gabriel
                        Höchstleistungsrechenzentrum Stuttgart
PCM

                                                                  Direct numerical
                                                                  simulation of turbulent
                                                                  reactive flows


                                                                  • 2-D flow fields
                                                                  • detailed chemical reactions
                                                                  • spatial discretization:
                                                                  6th order central derivatives
                                                                  • integration in time:
                                                                  4th order explicit Runge-Kutta

             Challenging Applications


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Requirements for a 3D simulation (I)

• Components
   – density, velocity and energy
   – mass fractions for chemical species (9 - 100)
• Spatial discretization
   – 100 m (typical flame-front)
   – 1 cm for computational domain
   – 100 grid-points into each direction
• Discretization in time
   – 10-8 for some important intermediate radicals
   – 1 s for slowly produced pollutants (e.g. NO)
 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Requirements for a 3D simulation (II)

• Summary
   – 100 variables
   – 106 grid-points
   – 1 ms simulation time with time steps of about
     10-8
  or
   – 105 iterations with 108 unknowns
• No such computation has been carried out up to
  now
 Grid-computing of interest to come closer to this
  goal

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Homogeneous Tests

• Comparison of execution for 10 iterations on one
  Cray T3E with Cray-MPI and with 2 Cray T3E and
  PACX-MPI
           Number of                         Execution time Execution time
           processes                         with Cray MPI  with PACX-MPI
                          4                         25.8 sec      28.3 sec

                         16                         25.9 sec      29.4 sec

                         32                         26.1 sec      32.0 sec

                         64                         26.0 sec      32.6 sec

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Heterogeneous Tests

• Execution time for 10 iterations on a Cray T3E and
  on an SGI Origin
       Number of                          Exectution time Execution time
       processes                          on Cray T3E     on SGI O2000
                    4                             25.8 sec        12.2 sec

                   16                             25.9 sec        14.4 sec

                   32                             26.1 sec           -

                   64                             26.0 sec           -

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Load-balancing

• Average and mean-value of execution times on
  each process calculated
• Transfer of complete lines only
• Relative load-imbalance used is calculated as


                                             | ti ( N )  t ( N ) |
                         L( N ) max
                                 i 1,..., p         t(N )



 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Influence of load-balancing




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Performance Analysis with MetaVampir

• Application consists of three parts
   – Pre-processing
   – Processing
   – Checkpointing of results

• Comparison of communication behaviour between
   – Cray MPI with 24 processors
   – PACX-MPI with 2 * 12 processors on 2 Cray
     T3E‟s


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Preprocessing MPI




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Preprocessing PACX-MPI




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Processing MPI




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Processing PACX-MPI




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Message statistics - process view




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Message statistics - cluster-view




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Message statistics - potential configuration




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Checkpointing of results with Cray-MPI




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Checkpointing of results with PACX-MPI




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Example of a production run

Cray T3E/900                                                      Hitachi SR8000
512 nodes                                                         16 nodes /128 CPUs
64 GByte memory                                                   128 GByte memory
                                    Hippi: 100MBit, 4ms



• auto-ignition process
• fuel (10%H2 and 90%N2, T=298K) and heated
  oxidiser (air at T=1298K)
• distribution is superimposed with turbulent flow-field
• temporal evolution computed

 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Maximum heat-release




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
More information about DAMIEN

• World Wide Web:
http://www.hlrs.de/organization/pds/projects/damien

• Publications:
Matthias Müller, Edgar Gabriel. Michael Resch:
A Software Development Environment for Grid-Computing
to be published in Concurrency and Computers - Practice
and Experience, Special Issue on Grid-Environments




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
The GrADS project


• To create an execution environment that supports
  reliable performance for numerical libraries on Grid
  computing platforms.
• Also include work on how to structure and optimize
  programs for execution on target computing
  platforms whose nature is not known until just
  before run time.




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
Milestones from GrADS


• Library interface for numerical libraries
• Infrastructure for defining and validating
  performance contracts
• Adaptive GrADS runtime interface for scheduling
  and forecasting systems in the Grid




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
ScaLAPACK Demo

• Implement a version of a ScaLAPACK library routine
  that runs on the Grid.
   – Make use of resources at the user‟s disposal
   – Provide the best time to solution
   – Proceed without the user‟s involvement
• Make as few changes as possible to the numerical
  software.
• Assumption is that the user is already “Grid enabled”
  and runs a program that contacts the execution
  environment to determine where the execution
  should take place.

  GFK 2 Workshop, Seoul                           Edgar Gabriel
                          Höchstleistungsrechenzentrum Stuttgart
Components Used

•   Globus version 1.1.3
•   Autopilot version 2.3
•   NWS version 2.0.pre2
•   ScaLAPACK version 1.6
•   ATLAS/BLAS version 3.0.2
•   BLACS version 1.1
•   MPIch-G version 1.1.2
•   PAPI 1.1.5

Millions of lines of code

    GFK 2 Workshop, Seoul                           Edgar Gabriel
                            Höchstleistungsrechenzentrum Stuttgart
Summary

• Tutorial covered several areas relevant for Grid-
  computing
• Focus on compute-intensive applications

• In case of any questions:
      gabriel@hlrs.de

• Slides are available at:
      http://www.hlrs.de/people/gabriel/tutorials


 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart
...and now good luck....




 GFK 2 Workshop, Seoul                           Edgar Gabriel
                         Höchstleistungsrechenzentrum Stuttgart

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:18
posted:9/2/2011
language:English
pages:186