Introduction to Parallel Computing Concepts by xld14276

VIEWS: 7 PAGES: 50

introduction-to-high-performance-computing pdf

More Info
									  High Performance Computing @ LSU - http://www.hpc.lsu.edu/




Introduction to Parallel 
 Computing Concepts


           Adam Jundt
     User Services Consultant
       HPC @ LSU, LONI



  Adam Jundt , User-Services Consultant, HPC @ LSU
            High Performance Computing @ LSU - http://www.hpc.lsu.edu/


                          Outline
• Motivation What is parallel computing, who’s using 
  it, and why.
• Computer Hardware How the hardware attributes 
  have an impact on the style and structure of 
  programs
• Parallel Software Parallelization of serial programs 
  will be described, with emphasis on MPI and 
  OpenMP



            Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



Supercomputing Power




                       Source: 
                       http://en.wikipedia.org/wiki/File:PPTSuperComputersPRINT.jpg
 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



               Top 500 List




 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/




 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/




 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



         Supercomputers




 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



    Supercomputing OS




             Source: 
             http://en.wikipedia.org/wiki/File:Operating_systems_used_on_top_500_supercomputers.svg
 Adam Jundt , User-Services Consultant, HPC @ LSU
         High Performance Computing @ LATech - http://www.hpc.lsu.edu/



             What is HPC About?

• Size: Many problems that are interesting to 
    scientists and engineers can’t fit on a PC – 
    usually because they need more than a few 
    GB of RAM, or more than a few 100GB of disk
• Speed: Many problems that are interesting to 
    scientists and engineers would take a very 
    long time to run on a PC: months or even 
    years. But a problem that would take a month 
    on a PC might only take a few hours on a 
    supercomputer
          Adam Jundt , User-Services Consultant, HPC @ LSU
           High Performance Computing @ LATech - http://www.hpc.lsu.edu/



 What are People Using HPC Doing?
• Simulation of physical phenomena
      • Weather forecasting
      • Black holes colliding
      • Oil reservoir management
• Data mining: finding needles of information in a 
    haystack of data
      • Gene sequencing
      • Signal processing
      • Detecting storms that could produce tornadoes
• Visualization: turning lots of data into pictures that a 
    scientist can understand.
             Adam Jundt , User-Services Consultant, HPC @ LSU
            High Performance Computing @ LATech - http://www.hpc.lsu.edu/


       “Traditional” Supercomputing 
                Architecture
 




    Interconnect network is on the order of 10Gb/s bandwidth,
    and accounts for ~30% of the cost of the supercomputer.




              Adam Jundt , User-Services Consultant, HPC @ LSU
  High Performance Computing @ LATech - http://www.hpc.lsu.edu/


“Traditional” Supercomputing 
     Architecture Cont’d




   Adam Jundt , User-Services Consultant, HPC @ LSU
          High Performance Computing @ LATech - http://www.hpc.lsu.edu/



     Other Implementations of HPC

• Clusters: A cluster of PCs uses low­cost PCs or 
    moderate cost ‘server’ computers available for 
    the mass market that are connected with 
    conventional Ethernet or a specialized 
    communication network.
      • Should have administration software such as a batch 
          queue system to be considered a cluster. Usually 
          users can only log onto a few nodes, called “head 
          nodes” while the nodes doing computation are not 
          interrupted by servicing user requests such as 
          editing a file. 


           Adam Jundt , User-Services Consultant, HPC @ LSU
          High Performance Computing @ LATech - http://www.hpc.lsu.edu/



     Other Implementations of HPC

• Shared Memory: If more than one program can 
    change the same memory location directly, 
    typically memory on the same motherboard, then 
    the architecture is call SMP.
• Distributed Computing: Using resources that 
    aren’t physically co­located to do work (more on 
    this later).




           Adam Jundt , User-Services Consultant, HPC @ LSU
         High Performance Computing @ LATech - http://www.hpc.lsu.edu/



                         HPC Issues

• Bandwidth: The amount of data transferred per 
    second. 
• Latency: The time delay beyond what is 
    intrinsic to the bandwidth limit. For example, 
    the time taken by the header of a message to 
    travel between two directly connected nodes 
    in the network. 




          Adam Jundt , User-Services Consultant, HPC @ LSU
                   High Performance Computing @ LATech - http://www.hpc.lsu.edu/



                          HPC Issues Cont’d
•   Registers
          •     Bandwidth: 351 GB/s
          •     Latency: ~1 CPU Cycle
•   Cache
          •     Bandwidth: 253 GB/s
          •     Latency: ~10 CPU Cycles 
•   Main Memory
          •     Bandwidth: 10.6 GB/s
          •     Latency: 15­20 ns (200­600 CPU cycles)
•   Infiniband Interconnection
          •     Bandwidth: 10­20 Gb/s
          •     Latency: 5­10 us
•   Hard Disk
          •     Bandwidth: 50­125 MB/s sustained data rate
          •     Latency: 3.5­5.0 ms seek time

                     Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



                 Parallelism




 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



 Jigsaw Puzzle Analogy




 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



        Serial Computing




 Adam Jundt , User-Services Consultant, HPC @ LSU
 High Performance Computing @ LATech - http://www.hpc.lsu.edu/



Shared Memory Parallelism




  Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



  The More the Merrier?




 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



     Diminishing Returns




 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



  Distributed Parallelism




 Adam Jundt , User-Services Consultant, HPC @ LSU
  High Performance Computing @ LATech - http://www.hpc.lsu.edu/



More Distributed Processing




   Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



           Load Balancing




 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



   More Load Balancing




 Adam Jundt , User-Services Consultant, HPC @ LSU
         High Performance Computing @ LATech - http://www.hpc.lsu.edu/



       Even More Load Balancing

• Three aspects to load balancing
     • Balancing data (memory)
     • Balancing work (computation)
     • Balancing interaction (communication)




          Adam Jundt , User-Services Consultant, HPC @ LSU
           High Performance Computing @ LATech - http://www.hpc.lsu.edu/



             Programming for HPC

• Currently, there are two main types of supercomputers 
• Clusters, where the memory is distributed with the 
    processors
      • Usually programmed with message passing (MPI)
• Shared memory systems, where all processors can 
    access all memory directly
      • Often programmed with MPI
      • Can also use threads
               • Posix: pThreads
               • OpenMP: standard for shared memory systems


            Adam Jundt , User-Services Consultant, HPC @ LSU
         High Performance Computing @ LATech - http://www.hpc.lsu.edu/



                    HPC Compilers

• Goal: create compilers that automatically 
   parallelize applications
     • Hasn’t worked yet, after about 20 years of trying
• HPC compilers are often standard compilers 
   (Fortran, C, C++) with some extra features or 
   extra libraries
     • OpenMP uses compiler directives to decide what 
        to parallelize
     • MPI uses libraries


          Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



Programming in OpenMP




 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



      A Team of Threads




 Adam Jundt , User-Services Consultant, HPC @ LSU
      High Performance Computing @ LATech - http://www.hpc.lsu.edu/



Splitting a Loop into Parallel Threads




       Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



       Trace of Execution




 Adam Jundt , User-Services Consultant, HPC @ LSU
  High Performance Computing @ LATech - http://www.hpc.lsu.edu/



Shared and Private Variables




   Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



         Race Conditions




 Adam Jundt , User-Services Consultant, HPC @ LSU
  High Performance Computing @ LATech - http://www.hpc.lsu.edu/



Critical Regions with OpenMP




   Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



    Programming in MPI




 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



             MPI Overview




 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



     MPI Send and Recv




 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



       Example Program




 Adam Jundt , User-Services Consultant, HPC @ LSU
  High Performance Computing @ LATech - http://www.hpc.lsu.edu/



Creating a Parallel Algorithm




   Adam Jundt , User-Services Consultant, HPC @ LSU
         High Performance Computing @ LATech - http://www.hpc.lsu.edu/



  Finding Concurrent Pieces of Work

• Decomposition:
     • The process of dividing the computation into 
         smaller pieces of work, i.e. tasks
• Tasks are programmer defined and are 
    considered to be indivisible




          Adam Jundt , User-Services Consultant, HPC @ LSU
  High Performance Computing @ LATech - http://www.hpc.lsu.edu/

Example: Dense Matrix­Vector 
       Multiplication




   Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



       Scientific Libraries




 Adam Jundt , User-Services Consultant, HPC @ LSU
         High Performance Computing @ LATech - http://www.hpc.lsu.edu/



    Other Implementations to HPC

• Distributed Computing
     • Using resources that aren’t physically co=located 
        to do work
• Examples:
     • Folding@home – Understanding protein folding, 
         misfolding, and related diseases
     • SETI@home – Search for Extraterrestrial 
         Intelligence




          Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



              SETI@home




 Adam Jundt , User-Services Consultant, HPC @ LSU
High Performance Computing @ LATech - http://www.hpc.lsu.edu/



      How Does it Work?




 Adam Jundt , User-Services Consultant, HPC @ LSU
          High Performance Computing @ LATech - http://www.hpc.lsu.edu/



       Why Bother with HPC at All?

• It’s clear that making effective use of HPC takes 
     quite a bit of effort, both learning how and 
     developing software
• That seems like a lot of trouble to go to just to 
     get your code to run faster
• It’s nice to have a code that used to take a day 
     to run finish in an hour. But if you can afford to 
     wait a day, what’s the point of HPC?
• Why go to all that trouble just to get your code 
     to run faster?
           Adam Jundt , User-Services Consultant, HPC @ LSU
         High Performance Computing @ LATech - http://www.hpc.lsu.edu/



     Why HPC is Worth the Bother

• What HPC gives you that you won’t get 
   elsewhere is the ability to do bigger, better, 
   more exciting science. If your code can run 
   faster, that means that you can tackle much 
   bigger problems in the same amount of time 
   that you used to need for smaller problems
• HPC is important not only for its own sake, but 
   also because what happens in HPC today will 
   be on your desktop in about 15 years: it puts 
   you ahead of the curve.
          Adam Jundt , User-Services Consultant, HPC @ LSU
                  High Performance Computing @ LATech - http://www.hpc.lsu.edu/



                                          Resources
•   Dr. A. Scheinine – Slides taken from his talk on Introduction to Parallel 
        Programming Concepts
         •   All of his references are listed here: http://www.cct.lsu.edu/~scheinin/Parallel/

•   B. Estrade – Slides taken from his talk on Introduction to OpenMP
         •   http://www.hpc.lsu.edu/training/20081022/presentations/introtoopenmpbde.pdf

•   Dr. D. Katz – Slides taken from his talk on Introduction to High Performance 
        Computing
         •   http://www.hpc.lsu.edu/training/20071127/HPC_intro.pdf

         •   Used many slides from Henry Neeman, University of Oklahoma, 
               “Supercomputing in Plain English”
•   Dr. G. Karypis – Slides taken from class on Introduction to Parallel 
        Computing
•   Dr. Thomas Sterling – Slides from “Scaling to Petascale: Concepts and 
        Beyond, Scaling to the Present and Future”, Virutal School for 
        Computational Science and Engineering, August 2009.
                    Adam Jundt , User-Services Consultant, HPC @ LSU

								
To top