Collaborative Analysis Toolkit

Document Sample
Collaborative Analysis Toolkit Powered By Docstoc
					   Collaborative Analysis Toolkit
                for
       Partial Wave Analysis

                reported by Richard Jones
GlueX collaboration meeting, Newport News, May 13, 2009
                                  GlueX collaboration meeting, Newport News, May 13, 2009




 GOAL: Develop a framework for amplitude analysis that
  is modular and independent of experiment.

  scales to very large data sets.
                                                     GRID
  accommodates increased computational demands.

  allows amplitudes to be written by the physicist.

  encourages a closer theory-experiment collaboration.




courtesy of Ryan Mitchell, Munich PWA workshop
                                  GlueX collaboration meeting, Newport News, May 13, 2009




 COLLABORATION:
    Funded by the National Science Foundation (NSF)
     Physics at the Information Frontier (PIF) program.
    Indiana University: R. Mitchell, M. Shepherd, A.
     Szczepaniak
    Carnegie Mellon University: C. Meyer, M. Williams
    University of Connecticut: R. Jones, J. Yang
    Plus: 2 more postdocs




courtesy of Ryan Mitchell, Munich PWA workshop
                                            GlueX collaboration meeting, Newport News, May 13, 2009



Overview: work flow
        4-Vector
          Files           3.Kinematic                         Rules          Methods
                            Variable          Kinvar
               Describe    Generation          files

 Generate
                           2.Amplitude                       5.Build a Fit     6.Run a Fit
            Data                              Amp
                            Generation
            Events                            files



    3 directories:        4.Normalization     Norma-
    data, acc, raw            Integral
                                              lization
                             Generation
                                                files
                                   GlueX collaboration meeting, Newport News, May 13, 2009




                  Layout of the Framework




   Highlights for this talk:
      (1) Distributed Data and Computing.
      (2)TheAmplitude Interface.

                                                                                    9
courtesy of Ryan Mitchell, Munich PWA workshop
                                  GlueX collaboration meeting, Newport News, May 13, 2009




courtesy of Ryan Mitchell, Munich PWA workshop
                                  GlueX collaboration meeting, Newport News, May 13, 2009




courtesy of Ryan Mitchell, Munich PWA workshop
                                  GlueX collaboration meeting, Newport News, May 13, 2009




courtesy of Ryan Mitchell, Munich PWA workshop
                                  GlueX collaboration meeting, Newport News, May 13, 2009




courtesy of Ryan Mitchell, Munich PWA workshop
                         GlueX collaboration meeting, Newport News, May 13, 2009



Ongoing work on 3p system: Peng et.al.
                                    GlueX collaboration meeting, Newport News, May 13, 2009




courtesy of Hrayr Matevosyan, Indiana University
                                    GlueX collaboration meeting, Newport News, May 13, 2009




courtesy of Hrayr Matevosyan, Indiana University
                                    GlueX collaboration meeting, Newport News, May 13, 2009




courtesy of Hrayr Matevosyan, Indiana University
                               GlueX collaboration meeting, Newport News, May 13, 2009



Ongoing work at Connecticut
 1. Porting the Ruby-PWA toolkit to the grid
  1.1 Install the Ruby-PWA software bundle on the UConn site
  1.2 Benchmark and scaling studies
  1.3 Parallel Ruby-PWA
 2. Integration of grid storage into Ruby-PWA framework
  2.1 Upgrade to the latest version of RedHat Linux
  2.2 Monitoring data integrity of dCache pools
  2.3 Grid data storage platform benchmarks
 3. Extending the UConn cluster for use as a grid testbed
  3.1 Installation
  3.2 Tuning




                                                                                   14
                                   GlueX collaboration meeting, Newport News, May 13, 2009




1.1 Install the Ruby-PWA software bundle on the
  UConn site
 ◦ Ruby-PWA
    A flexible framework for doing partial wave analysis fits
    Written in Ruby
    C++ classes wrapped as Ruby objects
 ◦ Installation
      Prerequisite: ruby-minuit
      Updated version of crenlib root package
      A number of build errors had to be investigated and solved
      Run Ruby-PWA test fits




                                                                                       15
                                     GlueX collaboration meeting, Newport News, May 13, 2009




1.2 Benchmark and scaling studies
 ◦ Test data bundle from CMU
 ◦ Original test data: 2188 simulated data events
 ◦ Create test samples of size x10, x20, x30, and x50 by duplicating Original
   test data
 ◦ More extensive scaling studies with statistically independent data will be
   carried out once the code is ported to a parallel platform.
                       Run Time(sec.) vs. Number of Events
            900

            800

            700
                                   y = 0.0332x2 + 11.68x + 1.79
            600

            500

            400                                                                RunTime

            300                                                                Poly. (RunTime)

            200

            100

              0
                                                                            Events* 2188
                  0    10     20          30         40           50   60                        16
                                  GlueX collaboration meeting, Newport News, May 13, 2009




1.3 Parallel Ruby-PWA
 ◦ MPI 2.0 new features
    Dynamic process management
      "the ability of an MPI process to participate in the creation of new MPI
       processes or to establish communication with MPI processes that
       have been started separately.“ – From MPI-2 specification
    One-sided communication
      Three one-sided communications operations, Put, Get, and
       Accumulate, being a write to remote memory, a read from remote
       memory, and a reduction operation on the same memory across a
       number of tasks
      Three different methods for synchronizing this communication -
       global, pairwise, and remote locks.
    Collective extensions
      In MPI-2, most collective operations apply also to
       intercommunicators


                                                                                      17
                                 GlueX collaboration meeting, Newport News, May 13, 2009




1.3 Parallel Ruby-PWA
 ◦ Comparison of OpenMPI 1.3 and MPICH 2
    OpenMPI 1.3
      Open MPI is an open source, production quality MPI-2
       implementation
      Developed and maintained by a consortium of academic, research,
       and industry partners
      OpenMPI design is centered around component concepts.
      Network connection devices: Shared memory, TCP, Myrinet, and
       Infiniband
      Network connection devices are dynamically selected in run time.
      We have tested this feature of OpenMPI. We also tested other basic
       functions of MPI, and got good performance
      We are now incorporating OpenMPI into Condor




                                                                                     18
                                GlueX collaboration meeting, Newport News, May 13, 2009




1.3 Parallel Ruby-PWA
 ◦ Comparison of OpenMPI 1.3 and MPICH 2(cont.)
    MPICH 2
      High-performance implementations of MPI-1 and MPI-2 functionality
      MPICH2 separates communication from process management.
      MPICH 2 channel devices such as sock, ssm, shm, etc…, can be
       specified on installation.
    We select OpenMPI for future development of ruby-pwa
      Advanced component architecture
      Dynamic communications features
      Broad support from academic and industry partners.




                                                                                    19
                                 GlueX collaboration meeting, Newport News, May 13, 2009




1.3 Parallel Ruby-PWA
 ◦ Hybrid programming model : OpenMP + MPI
    OpenMP defines an API for writing multithreaded applications for
     running specifically on shared memory architectures.
    Greatly simplifies writing multi-thread programs in Fortran, C and C++.
    gcc version 3.4 and above on Linux supports OpenMP.
    The hybrid programming model, OpenMP + MPI, combines the shared-
     memory and distributed-memory programming models.
    In our tests, OpenMP implementation ran very efficiently on up to 8
     processes.
    OpenMPI implementation ran with essentially 100% scaling, provided
     that all of the communications channels were tcp-ip sockets.
    OpenMPI tests using a mixture of shared-memory and tcp-ip
     communcations channels showed markedly lower performance. We are
     still investigating its cause.



                                                                                     20
                                GlueX collaboration meeting, Newport News, May 13, 2009




1.3 Parallel Ruby-PWA
 ◦ GPU Computing
    From Graphics Processing to Parallel Computing
      GPU (Graphic Processor Unit) has evolved into a highly parallel,
       multithreaded, manycore processor with tremendous computational
       horsepower and very high memory bandwidth
      In November 2006, NVIDIA introduced CUDA™, a general purpose
       parallel computing architecture – with a new parallel programming
       model and instruction set architecture.
      The initial CUDA SDK was made public 15 February 2007. NVIDIA has
       released versions of the CUDA API for Microsoft Windows, Linux and
       Mac OS X.
    Our test platform
      GeForce 9800 GT(14 multiprocessors, 112 cores, 512MB memory)
      Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz
      8GB memory


                                                                                    21
                                          GlueX collaboration meeting, Newport News, May 13, 2009




1.3 Parallel Ruby-PWA
 ◦ GPU Computing (cont.)
    Benchmark: Matrix Multiplication

            Matrix Multiplication Run Time , GPU vs CPU (ms)
    16000                                              15065

    14000

    12000

    10000

     8000                                                        GPU

                                                                 CPU
     6000

     4000

     2000
                    54              435
              0.9               5                 30
        0
              256x256          512x512           1024x1024



                                                                                              22
                                     GlueX collaboration meeting, Newport News, May 13, 2009




1.3 Parallel Ruby-PWA
 ◦ GPU Computing (cont.)
    Constraints of GPU computing
      One GPU can only be used by one process, it can not be shared by other
       processes.
      Developers should design delicate programs to parallel their applications to
       overlap the latency of global memory.
      CUDA uses a recursive-free, function-point-free subset of C language.
      The programming skill for CUDA is new to developers. The learning cure may
       be steep for developers.
      In some applications, it’s difficult for CUDA to achieve so high performance as
       matrix multiplication does.
    Challenges
      How to incorporate GPU computing into paralleling Ruby-PWA
      How to explore, share and manage GPU resources on grid
      How to hide the complexity of GPU computing to developers




                                                                                         23
                           GlueX collaboration meeting, Newport News, May 13, 2009




2.1 Upgrade to the latest version of Redhat Linux
 ◦ Upgrade operating systems to latest version of Redhat
   Linux (CentOS 5 distribution)
 ◦ Upgrade Linux kernel to 2.6.26
 ◦ The upgrades are deployed to more than 80 nodes on
   UConn site.




                                                                               24
                                 GlueX collaboration meeting, Newport News, May 13, 2009




2.2 Monitoring data integrity of dCache pools
 ◦ dCache
      Single 'rooted' file system name space tree
      Supports multiple internal and external copies of a single file
      Data may be distributed among a huge amount of disk servers.
      Automatic load balancing by cost metric and interpool transfers
      Distributed Access Points (Doors)
      Automatic HSM migration and restore
      Widely used at Tier II and Tier III centers in the LHC data grid.
 ◦ We have adopted dCache as the network data storage
   infrastructure at UConn site
 ◦ We developed scripts to monitor the integrity for all files
   stored in dCache pools


                                                                                     25
                                GlueX collaboration meeting, Newport News, May 13, 2009




2.3 Grid data storage platform benchmarks
 ◦ Mechanisms for reading data stored in dCache pools
    Similar methods to unix streams: open, read blocks of data, and
     close.
    File transfer program like rft.
 ◦ Protocols
    dCap: dCap API provides an easy way to access dCache files as
     if they were traditional unix stream files
    gsidCap: secure version of dCap
    gridftp: the standard protocol to transfer files on the grid
     platform.
 ◦ Performance test
    Open a file, read one block of data, and close the file.
    600 files tested


                                                                                    26
                              GlueX collaboration meeting, Newport News, May 13, 2009




2.3 Grid data storage platform benchmarks
 ◦ dCap
    Avg. 690ms for access one block of a file




                                                                                  27
                              GlueX collaboration meeting, Newport News, May 13, 2009




2.3 Grid data storage platform benchmarks
 ◦ gsidCap
    Avg. 750ms for access one block of a file




                                                                                  28
                                 GlueX collaboration meeting, Newport News, May 13, 2009




2.3 Grid data storage platform benchmarks
 ◦ GridFTP
    Avg. 1700ms for access one block of a file




                                                                                     29
                                       GlueX collaboration meeting, Newport News, May 13, 2009




3.1 Installation
     New cluster is added to UConn physics grid platform
          32 DELL PowerEdge 1435 servers
          quad core processors per node
          8 GB memory per node
          One 320GB hard disk per node
          Two gigabit network interfaces per node
3.2 Tuning
  ◦ To reduce the cost of maintenance, we use NFS root file system
  ◦ Tune TCP settings to make the new cluster stable




                                                                                           30
GlueX collaboration meeting, Newport News, May 13, 2009




                                                    31

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:24
posted:2/25/2012
language:
pages:31