Docstoc

Klasky

Document Sample
Klasky Powered By Docstoc
					 Fusion End-to-End Computing on
 Leadership Class Computers.




                   SC06 Workshop on Ultra-Scale
                          Visualization
                                   11/13/2006
                             Scott A. Klasky (ORNL)
                             Stéphane Ethier (PPPL)

OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov                                       1
         Outline
          The End-to-end framework.
          The Fusion Applications.
                   GTC.
                   XGC/M3D/Nimrod.
                 Kepler Workflow Automation.
                 Data In Transit.
                 Visualization.
                 Ubiquitous and Transparent Data Access.
                 Dashboard.
                 Conclusions.



OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
  It’s all about the enabling technologies…




                                 Applications


                                    Math
       Applications                             Enabling
              drive                  CS
                                                technologies
D. Keyes                                        respond
 OAK RIDGE NATIONAL LABORATORY
 U. S. DEPARTMENT OF ENERGY
 klasky@ornl.gov
         The End-to-End Framework




                                                                   Metadata rich output from components.
                                  Applications
                                                 VIZ/Dashboard
                     Applied Math


                  Workflow Automation
                                                 Data Monitoring
                   CCA

              SRM            LN       Async. NXM streaming
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         Enabling Technology Areas



         It’s all
          Applications (GTC, XGC, M3D, S3D)
                   S. Ethier, S. Ku, J. Cummings, G.Y. Park, R. Sankaran
          Kepler Workflow framework.
                   N. Podhorszki, A. Shoshani, I. Altintas, B. Ludaescher, M. Vouk
          Code Coupling Framework/Data Transfer


          about the
                   C. Docan, M. Parashar, K. Schwan, M. Wolf
          Visualization.
                   K. Bennett, E. Feibush, A. Kahn, K. Ma, S. Parker, D. Silver
          Ubiquitious and Transparent Data Access.
                   A. Shoshani, A. Sim, M. Beck, C. Sellers


          data!
          Dashboard
                   R. Barreto, M. Vouk


OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
Hardware architecture (now)
                  Lustre
             Restart files
                                                                               HPSS
                                           Jaguar
                                             I/O
     Jaguar                                nodes
    compute                                                    Ewok
                           Simulation      40 G/s    Sockets
     nodes                   control                  later
        Login
       nodes
                                                    Lustre2           Nfs DB



                             Job control
                                                                  Ewok-web

                                        Ewok-sql
                                                                                Seaborg
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         Hardware in the near future.
          119 Tflop machine will give 1 Tflop for Real-
           time-analysis.
                   Run service node linux on n-compute nodes.
                   Fast bandwidth between 2 systems.
                   Allows for sockets + threads + full linux.
          We will have a machine(s) for running a web
           server.
          We will have a machine for the sql server.
          For the Pflop machine, we will use 100Tflop
           for real-time-analysis.
                   250 Tflop machine will be the post-processing
                    cluster.
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         The Applications.
          Focus on leadership class applications.
                     Typical team(physics) side <5 people.
                     Typical Simulations use >50% of available processors.
                     Typical Simulations run >10 hours.
                     1 Large Simulation/(1-2 months).
                     Typical Data Sets are 10% of available memory per time step dump.
                     Typical output is >2TB/simulation (today).
                         100TB of data for Pscale simulations!
          Need to focus on
                   I/O: Apps will not write out much data if I/O cost are too high. <5%
                   Data Movement: Need to move data across the country. (Fusion
                    data needs to be shared!)
                   Simulation Monitoring.
                   Web-based Visualization.
                   High-End Visualization for “finding the needle in the haystack”
                          Feature Tracking/Detection of highly turbulent data.
                          High Dimensional (5D+time) particle visualization (1x1012)
                          Visualization for code coupling.
                          Puncture Plot feature detection.

OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
   GPSC gyrokinetic PIC codes used for studying
   microturbulence in plasma core
                                                           GPSC
 GTC (Z. Lin et al., Science 281, p.1835, 1998)
   Intrinsically global 3D nonlinear gyrokinetic PIC code
                                                           SciDAC
          All calculations done in real space
          Scales to > 10,000 processors
          Delta-f method
          Currently being upgraded to fully electromagnetic
 GEM (Y. Chen & S. Parker, JCP, in press 2006)
                                                                        CPES
          Fully electromagnetic nonlinear delta-f code                 SciDAC
          Split-weight scheme implementation of kinetic electrons
          Multi-species
          Uses Fourier decomposition of the fields in toroidal and poloidal
           directions (wedge code)



OAK RIDGE NATIONAL LABORATORY                   Date
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
                              GTC Code performance.




                                                                                                   GTC/XGC
                   10000               GTC Data Generation
                    1000
                                                                               Increase output because of
                    100                Tflops                                  •Asynchronous metadata rich I/O.
                     10



                      1
                                                                               •Workflow automation.
                     0.1
                                                      TB                       •More analysis services in the
                    0.01
                                                                               workflow.
                       1996   1998   2000   2002   2004   2006   2008   2010


OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         1998: Nonlinear zonal flow simulations by
         GTC with 108 particles on Cray T3E
                 Nonlinearly generated zonal flows
                  associated with ITG turbulence
                  have been observed to break up
                  the eddies and reduce transport
                  in global simulations [Lin, Hahm,
                  Lee, Tang, White, Science 1998]




OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         And Today
                                Image generated from
                                workflow.
                                Users need to set angles
                                (min/max) and then
                                workflow produces the
                                images.
                                Still need to put this in
                                our GTC monitoring for
                                everyday use!




OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         2002: First global ITER-size simulation using
         109 particles on IBM SP 3
                                                   Size of eddies ~constant
          Transition from Bohm to GyroBohm
           scaling was finally observed and the
           elusive GyroBohm scaling is finally
           found [Lin, Ethier, Hahm, Tang,
           PRL2002]
          Data Streaming Technology to move
           terabytes of data from NERSC to PPPL
           [Klasky, Ethier, Lin, et al., SC2003]



    Good news for
        ITER!




OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         GTC simulations show that turbulence
         spreading causes Bohm scaling

  GTC: Turbulence spreading is
   responsible for transport Bohm
   scaling in small devices
   [Lin, Hahm, PoP2004]
  ITG simulations using GTC for shaped
   plasmas have confirmed the nonlocal
   property of turbulent transport
   [Wang et al., PoP2006]




OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
          Comparison of particle flow
           Particles diffuse faster with the velocity space non-linearity.
                   Allows simulations to produce correct results faster.
                   Visualization shows movement of particles around the eddies.
           Really require 5D+time phase space viz.
                   40x109 particles now.
                   1x1010 particles in future, with larger machine size.
           Comparison visualization is a necessary tool for scientific discovery.




                                           QuickTime™ an d a
                                      MPEG-4 Video decompressor
                                     are need ed to see this p icture .




OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         4x1010 particles on NCCS Cray XT3 enables
         stringent convergence test of particle noise

  Particle noise-induced
   transport is 1000
   times smaller than
   electron temperature
   gradient (ETG)
   turbulence transport
  ETG saturates via
   nonlinear toroidal
   coupling [Lin et al,
   PoP2005]. Turbulence


                                  c (reve2/LT)
                                                                ETG flux     Track flow
   spreading, zonal                                                          across
   flows, and nonlinear                                                      individual
   phase space structure                                                     eddies.
   all regulate the                                             Noise flux   Visualization
   dynamics of                                                               showed
   ETG streamers                                 Time (LT/Ve)


OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
 Advanced data analysis techniques using
 principal component analysis
                  1t
                  2t

                      ƒ (Xt) = S0 + 1t S1 + … + kt Sk + Et
                                                       S1
                                                                        Transform to reduce
                                                                         non-linearity in distribution
                                                                         (often density-based)
                                                                        PCA computed via SVD
                                                                         (or ICA, FA, etc.)
                                                                        Construction
                                                                         of component movies
                                                                        Interpretation of spatial, time,
                                                                         and movie components
                                                     Ostrouchov ORNL
                   EETG GTC Simulation Data
                   (W. Lee, Z. Lin, and S. Klasky)                      Pairs of equal singular values
                   Decomposition shows transient                         indicate periodic motion
                   wave components in time

OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         Physics in tokamak plasma edge
   Plasma turbulence
   Turbulence suppression (H-mode)
   Edge localized mode and ELM cycle
   Density and temperature pedestal
   Diverter and separatrix geometry
   Plasma rotation
   Neutral collision




                                                       ITER (www.iter.org)
                             Diverted magnetic field
Edge turbulence in NSTX
(@ 100,000 frames/s)
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         XGC 1 code
     Particle-in-cell code
     5-dimensional (3-D real space
       +
       2-D velocity space)
     Conserving plasma collisions
       (Monte Carlo)
     Full-f ions, electrons, and
       neutrals
     Gyrokinetic Poisson equation
       for neoclassical and turbulent
       electric field
     PETSc library for Poisson
       solver
     MPI for parallelization
     Realistic magnetic geometry
       containing X-point
     Particle source from neutral
       ionization
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         Neoclassical potential and flow of
         edge plasma from XGC1




                Electric potential   Parallel flow and particle positions
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         XGC-MHD coupling plan
    •Need real-time visualization to help monitor/debug these simulations.
                                        XGC-0 grows pedestal along
    •Need better integration with TotalView (or otherroot
                                        neoclassical debugger).
     Phs-0: Simple coupling:
    •Need to be able to look at derived quantities from raw data.
                                        MHD checks instability and crashes
     with M3D and NIMROD                the pedestal

                                     The same with XGC-1 and 2

      Phs-2: Kinetic coupling:       XGC supplies closure information to
      MHD performs the crash         MHD during crash


      Phs-3: Advanced coupling:      M3D supplies the B crash
      XGC performs the crash         information to XGC during the crash



                  Black: Developed • Red: To be developed
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
     XGC-M3D code coupling
     Code coupling framework with
     Kepler-HPC
                                               End-to-end system 160p, M3D
       XGC on Cray XT3                         runs on 64P
                                               Monitoring routines here

                                     40 Gb/s




                         User monitoring                                     Data replication




                                                       Data replication




                                                       Post-processing




                                                                                         Ubiquitous and
                                                                               transparent data access
OAK RIDGE NATIONAL LABORATORY                                                  via logistical networking
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
M3D equilibrium and linear simulations
new equilibrium from eqdsk, XGC profiles




              Equilibrium poloidal   Linear perturbed poloidal      Linear perturbed
                 magnetic flux          magnetic flux, n = 9     electrostatic potential
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
     Kepler workflow framework
   Kepler: developed by the SDM
     Center
         Kepler is an adaptation of the UC Berkeley
          tool, Ptolemy
         Can be composed of sub-workflows
         Uses event-based “director” and “actors”
          methodology
         Features in Kepler relevant to CPES
                 Launching components (ssh, command
                  line)
                 Execution logger – keep track of runs
                 Component wrapping – based on CCA
                 Data movement – Sabul, Gridftp,
                  Logistical Networks (future), data
                  streaming (future).




OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         Workflow Automation. Current-Status.
       Kepler provides the glue for the fusion workflows.
       Successfully developed workflow for CPES.
         Now used for GTC, S3D, + many more.
         Main Part of the Kepler Tutorial at SC2006!
       Work Accomplished.
                 permanent SSH connection to machines secured by OTP
                 basic job submission capability to several resource
                  management systems: PBS, LoadLeveler, SGE, Condor.
                 New Kepler actors for the monitoring pipeline defined in CPES
                 Watching a directory for new files (generated by a simulation)
                 Generic actor to process a file
                      e.g. transfer it to another machine using scp or bbcp
                      execute bp2h5 bp  hdf5 converter
                 Archiver actor to pack many files together and tar to HPSS.
                 Development of the processing workflow for transferring simulation files,
                  conversion and archival.
                 Kepler install and maintenance on ewok.
                 Kepler HOWTO.

OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         Workflow in Action


                                                           Watch




                                     Transfer
                                           Convert
                                                     Archive


                   Transfer from NERSC to ORNL, 3000 timesteps,
                    800GB within the simulation run (34 hours)
                   Convert each file to HDF5 file
                   Archive files to 4GB chunks to HPSS
                   All this performed by Stephane Ethier, PPPL
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         Workflow Automation: Next Steps.
         Monitor the simulations.
                  New code must be put into the HPC codes.
                  On the Dashboard, we choose which variables to
                   output, and the frequency (time steps) to output
                   the variables. (ewok-web)
                  ewok-web writes a file which contains variables
                   and frequency of output. (nfs directory).
                  Workflow transfers the file to lustre on jaguar.
                  In XGC (at the end of every timestep) new code
                   will open the monitor.dat file, and set variables
         Requirements for Petascale computing
                  Easy to use
                  – Dashboard front-end
                  – Autonomic

OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         Petascale data workspace




K. Schwan
M. Wolf
M. Parashar

OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
   Asynchronous petascale I/O for data in transit
                                                              High-performance
                                                               I/O
                                                                 Asynchronous
                                                                 Managed buffers
                                                                 Respect firewall
                                                                  constraints
                                                              Enable dynamic
                                                               control with flexible
                                                               MxN operations
                                                                 Transform using
                                                                  shared-space
                             User applications                    framework (Seine)
          Seine coupling framework interface

            Shared space                           Other
                               Load balancing
            management                            program
           Directory layer     Storage layer     paradigms
     Communication layer (buffer management)
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF EOperating system
                     NERGY
klasky@ornl.gov
         On the Path to Petascale Computing!
         Asynchronous I/O.
          Allows us to move massive amounts of data with very
           low overhead.
                   Create buffers on all of the computational nodes.
                   Memcpy or zero copy
                   Uses Remote Memory Access (RMA) via portals.
                   Expect <2% overhead writing/moving 20 TB/hour on Pflop
                    machine.
                   Allows us to move data off HPC without writing data.
                   Our I/O routines can be converted to use this very easily.
          Q: Where do we store all of this data?
                   One GTC simulation = (100 hours) * (20 TB) = 2 PB.
          Since our API’s go into codes, we can control the
           metadata!
                   Allows for better archiving of the data, more hints to the O/S.

OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         Visualization
             Monitoring Services.                      Post Processing.
      Current Status                               Current Status
          Use of AVS/Express as services into         Support Scientist needs
           Kepler.                                     New I/O routines for
          Use of SciRun as service into                visualization/post processing.
           Kepler.
                                                    Future Status
          Elvis Services.
                                                       Feature Tracking of ELM Blobs
      Future Status
          More visualization services                     Comparison of experimental
           integrated into workflow and                     to simulation data.
           dashboard.                                  Better Post processing
          Integration of post processing               environment.
           routines into dashboard.
          Feature tracking in workflow.
          AJAX, SVG, Elvis, Quicktime for
           monitoring dashboard.




OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
     Ubiquitous and transparent data sharing
                                                                 Petabytes

                                                                  Tapes

      IDAVE                                                        e.g. HPSS




                                                                       Terabytes
  IDAVE

                                                                       Disks

                                                                  Terabytes

  IDAVE
                                                                  Disks
                          Simulations and collaborators in any FSP will be
                           distributed across national and international networks
                IDAVE    FSP simulations will produce massive amounts of data
                           that will be permanently stored in national facilities, and
                           temporary stored at collaborators disk storage systems
                         Need
OAK RIDGE NATIONAL LABORATORY to share large volume of data amongst collaborators
U. S. DEPARTMENT OF ENERGY and the wider community.
klasky@ornl.gov
         Logistical Networking.
          Task:
                   When .bp files are converted to hdf5 files, placed in depots too
                      This returns an address (eXnode file). Access the data from
                       this eXnode file.
                      Need fast access to data.

                                                                    Directory
                                                                     server
HDF5 library fully operational
        Many apps ported by relinking
                                                                                                 NYU
        HDF5 utilities, bp2h5, AVS
        Express (soon)                                                                   PPPL
                                              Jaguar              Ewok
LoDN directory integrated w/ stdio            Cray              cluster
                                                                                                 MIT
& HDF5                                         XT3
        Open files stored on LoDN by URL              Portals

        Automatic distribution of data                                                    UCI
                                                                                Depots
        Distribution user-controlled per
        file/directory
Ongoing performance tuning
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         SRM-lite operation
         (HPSS to HPSS)
                                                           Anywhere
                                                           Kepler
                                                         Data Mover
                           Recovers from
                        file transfer failures
                                                           Actor                                  Recovers from
                                                                                                  staging failures
   Recovers from                                 SRM-COPY                      Initiate
  archiving failures
                          Initiate
                                                 (multiple files)

          ORNL                                                                                           LBNL
                                                      SRM-GET (one file at a time)
                       SRM-Lite                                                            SRM-Lite
                                                         SCP (pull mode)




                                  Disk                                                    Disk
                                  Cache                                                   Cache

                                                            Network transfer

             archive files                                                                   stage files
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         Dashboard front end
  Desktop Interface uses asynchronous
   Javascript and XML (AJAX); runs in
   web browser
  Ajax is a combination of technologies
   coming together in powerful ways
   (XHTML, CSS, DOM, XML, XST,
   XMLhttpRequest, and Javascript)
  The user’s interaction with the application
   happens asynchronously – independent of
   communication with the server
  Users can
   rearrange
   the page
   without
   clearing it
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         SDM/ORNL Dashboard: Current Status
 Step 1:
         Monitor ORNL and
          NERSC machines.
 Log in
            https://ewok-
             web.ccs.ornl.gov/dev/rbarreto/
             SDMP/WebContent/SdmpApp
             /home.php
         Uses OTP.
 Working to pull out
  users jobs.
 Workflow will need
  to move data to
  ewok web disk.
         Jpeg, xml files
          (metadata).

OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov
         Some Opinions.

          For Pscale computing, we need to integrate
           SDM with Visualization for real-time
           monitoring from remote locations.
                   Too much data to explore after the simulation.
                   Must be simple to put code into the framework.
                   Must allow for high performance data movement.
          Finding most of the “needles in the haystack”
           must be done as the data is being generated.
                   Limited manpower on the simulation scientist side.
          Query based post processing needs to be
           brought into all of our post processing tools!
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
klasky@ornl.gov

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:7
posted:11/9/2011
language:English
pages:37