Docstoc

GB GB complaint

Document Sample
GB GB complaint Powered By Docstoc
					     SOS7: “Machines Already Operational”
    NSF’s Terascale Computing System

            SOS-7 March 4-6, 2003
              Mike Levine, PSC

                                                                                




                                               T T       G
                                            P IT T S BU RG H
                                                          PUT N
                                            S U P ERC O M P U TIIN G

1                                           C    E N T E R
        Outline

     Overview of TCS, the US-NSF’s Terascale
      Computing System.
     Answering 3 questions:
          Is your machine living up to performance
           expectations? …
          What is the MTBI? …
          What is the primary complaint, if any, from users?
       [See also PSC web pages & Rolf’s info.]
                                                                                               




                                                              T T       G
                                                           P IT T S BU RG H
                                                                         PUT N
                                                           S U P ERC O M P U TIIN G

2                                                          C    E N T E R
     Q1: Performance

       Computational and communications
        performance is very good!
           Alpha processors & ES45 servers: very good
           Quadrics bw & latency: very good.
           ~74% of peak on Linpack; >76% on LSMS
     More work on disk IO.
     This has been a very ease “port” for most
      users.
           Easier than some Cray  Cray upgrades.                                           




                                                            T T       G
                                                         P IT T S BU RG H
                                                                       PUT N
                                                         S U P ERC O M P U TIIN G

3                                                        C    E N T E R
                    Q2: MTBI (Monthly Average)
    • Compare with theoretical prediction of 12 hrs.
    • Expect further improvement (fixing systematic problems).

                    14.0


                    12.0                                             11.3                  11.3                                                      11.1
    MTBI ( hours)




                                                                                  10.3                                      10.3
                                                                                                     10.1                               9.9
                    10.0                                                                                          9.4
                                                 8.7
                                                            8.0
                     8.0

                                    6.3
                     6.0


                     4.0


                     2.0


                     0.0
                                                                                                                                                                                                
                                                 2
                                 2




                                                                                2
                                                                    2




                                                                                                                  2




                                                                                                                                      3



                                                                                                                                                   3
                                                          2




                                                                                           2




                                                                                                                            2
                                                                                                     2
                               '0




                                                                              '0
                                                                  '0




                                                                                                                '0
                                              '0




                                                                                                                                                 '0
                                                        '0




                                                                                                                                   '0
                                                                                         '0




                                                                                                                          '0
                                                                                                  '0
                             ri l



                                          ay




                                                                ly




                                                                                                            ov
                                                                              g




                                                                                                                                                 b
                                                                                                                                  n
                                                                                      pt
                                                       ne




                                                                                                                      ec
                                                                                               ct
                                                                            Au




                                                                                                                                Ja



                                                                                                                                              Fe
                                                              Ju
                           Ap




                                                                                    Se


                                                                                               O
                                          M




                                                                                                            N




                                                                                                                                                               T T       G
                                                                                                                                                            P IT T S BU RG H
                                                     Ju




                                                                                                                      D




                                                                                                                                                                          PUT N
                                                                                                                                                            S U P ERC O M P U TIIN G

4                                                                                                                                                           C    E N T E R
    Time Lost to Unscheduled Events
                                        • Purple: nodes requiring cleanup
                                        • Worst case is ~3%
                                        4500
    Node Hours per Week (tot=126,000)




                                        4000

                                        3500

                                        3000

                                        2500

                                        2000

                                        1500

                                        1000

                                         500
                                                                                                                                                                                                         


                                           0
                                                                                                                    3




                                                                                                                                                                 3
                                                                  2




                                                                                                                               3


                                                                                                                                          3


                                                                                                                                                     3
                                               02


                                                        02




                                                                         02


                                                                                    02


                                                                                             02


                                                                                                       02


                                                                                                                 00




                                                                                                                                                              00
                                                               00




                                                                                                                             0


                                                                                                                                         0


                                                                                                                                                     0
                                            20


                                                      20




                                                                        20


                                                                                 20


                                                                                           20


                                                                                                    20




                                                                                                                          20


                                                                                                                                      20


                                                                                                                                                  20
                                                                                                               2




                                                                                                                                                            2
                                                             /2




                                                                                                                                                                        T T       G
                                                                                                                                                                     P IT T S BU RG H
                                                                                                            7/




                                                                                                                                                         4/
                                          9/


                                                    6/




                                                                      0/


                                                                               7/


                                                                                         4/


                                                                                                  1/




                                                                                                                           /


                                                                                                                                      /


                                                                                                                                                 /
                                                             /3




                                                                                                                        14


                                                                                                                                   21


                                                                                                                                              28
                                                                                                            1/




                                                                                                                                                         2/
                                          /1


                                                   /2




                                                                    /1


                                                                               /1


                                                                                        /2


                                                                                                  /3




                                                                                                                                                                                   PUT N
                                                                                                                                                                     S U P ERC O M P U TIIN G
                                                           12




                                                                                                                    1/


                                                                                                                                 1/


                                                                                                                                             1/
                                        11


                                                 11




                                                                  12


                                                                             12


                                                                                      12


                                                                                                12




5                                                                                                                                                                    C    E N T E R
        Q3: Complaints
       #1: “I need more time” (not a complaint about performance)
            Actual usage >80% of wall clock
            Some structural improvements still in progress.
            Not a whole lot more is possible!
       Work needed on
            Rogue OS activity.          [recall Prof. Kale’s comment]
            MPI & global reduction libraries.      [ditto]
            System debugging and fragility.
            IO performance.
               We have delayed full disk deployment to avoid data corruption &
                instabilities.
            Node cleanup
               We detect & hold out problem nodes until staff clean.
       All in all, the users have been VERY pleased.
                                                                                                                       




                                                                         [ditto]
                                                                                      T T       G
                                                                                   P IT T S BU RG H
                                                                                                 PUT N
                                                                                   S U P ERC O M P U TIIN G

6                                                                                  C    E N T E R
    Full Machine Job
       This system is capable of doing big science




                                                                                      




                                                     T T       G
                                                  P IT T S BU RG H
                                                                PUT N
                                                  S U P ERC O M P U TIIN G

7                                                 C    E N T E R
        TCS (Terascale Computing System) & ETF
        Sponsored by the U.S. National Science Foundation
        Serving the “very high end” for US academic computational science and
         engineering
             Designed to be used, as a whole, on single problems. (recall full machine job)
             Full range of scientific and engineering applications.
             Compaq AlphaServer SC hardware and software technology
             In general production since April, 2002
        #6 in Top 500; (largest open facility in the world: Nov 2001)
        TCS-1: in general production since April, 2002
        Integrated into the PACI program (Partnerships for Academic Computing
         Infrastructure)
             DTF project to build and integrate multiple systems
               – NCSA, SDSC, Caltech, Argonne. Multi-lamba, transcontinental interconnect
             ETF aka Teratrid (Extensible Terascale Facility) integrating TCS with DTF
              forming                                                                                                           




                – A heterogeneous, extensible scientific/engineering cyberinfrastructure Grid
                                                                                               T T       G
                                                                                            P IT T S BU RG H
                                                                                                          PUT N
                                                                                            S U P ERC O M P U TIIN G

8                                                                                           C    E N T E R
    Infrastructure: PSC - TCS machine room                   ( @ Westinghouse)
    (Not require a new building; just a pipe & wire upgrade; not maxed out)




                                                                          ~8k ft2
                                                                          Use
                                                                           ~2.5k
                                                                          Existing
                                                                           room.
                                                                          (16 yrs
                                                                           old.)


                                                                                                                  




                                                                                 T T       G
                                                                              P IT T S BU RG H
                                                                                            PUT N
                                                                              S U P ERC O M P U TIIN G

9                                                                             C    E N T E R
 Full System: Physical Structure
       CONTROL               DISKS




                                 SERVERS
                                           Floor Layout


                                                 Geometrical
                                                  constraints
                    SWITCH
                                                  invariant
                                                  twixt US &
                                                  Japan

                                                                                            




                 COMPUTE NODES
                                                          T T       G
                                                       P IT T S BU RG H
                                                                     PUT N
                                                       S U P ERC O M P U TIIN G

10                                                     C    E N T E R
     Terascale Computing System
                                  Compute Nodes
                                  • 750 ES45 4-CPU servers
                                      • +13 inline spares
                                      • (+2 login nodes)
                                  • 4 - EV68’s /node
               Compute Nodes
                                  • 1 GHz = 2.Gf     [6 Tf]
                                  • 4 GB memory [3.0 TB]
                                  • 3*18.2 GB disk [41 TB]
                                      • System
                                      • User temporary
                                      • Fast snapshots
                                           • [~90 GB/s]
                                  • Tru64 Unix
                                                                                                




                                                               T T       G
                                                            P IT T S BU RG H
                                                                          PUT N
                                                            S U P ERC O M P U TIIN G

11                                                          C    E N T E R
        ES45 nodes
            5 nodes per cabinet
            3 local disks /node




                                                                




                               T T       G
                            P IT T S BU RG H
                                          PUT N
                            S U P ERC O M P U TIIN G

12                          C    E N T E R
     Terascale Computing System
                                                      Quadrics Network
                                Quadrics
                                                     • 2 “rails”
                                                         • Higher bandwidth
                                                              • (~250 MB/s/rail)
                                                         • Lower latency
                                                              • 2.5 s put latency
                 Compute Nodes
                                                     • 1 NIC/node/rail
                                                     • Federated switch (/rail)
                                                     • “Fat-tree” (bbw ~0.2 TB/s)


          • User virtual memory mapped
          • Hardware retry
          • Heterogeneous
              • (Alpha Tru64 & Linux, Intel Linux)                                                                   




                                                                                    T T       G
                                                                                 P IT T S BU RG H
                                                                                               PUT N
                                                                                 S U P ERC O M P U TIIN G

13                                                                               C    E N T E R
     Central Switch Assembly

                                  20 cabinets
                                   in center
                                  Minimize max
                                   internode
                                   distance
                                  3 out of 4 rows
                                   shown
                                  21st LL switch,
                                   outside (not
                                   shown)


                                                                                  




                                                 T T       G
                                              P IT T S BU RG H
                                                            PUT N
                                              S U P ERC O M P U TIIN G

14                                            C    E N T E R
     Quadrics wiring overhead (view towards ceiling)




                                                                                     




                                                    T T       G
                                                 P IT T S BU RG H
                                                               PUT N
                                                 S U P ERC O M P U TIIN G

15                                               C    E N T E R
     Terascale Computing System
                                     Management & Control
      Control            Quadrics

     LAN                            • Quadrics switch control:
                                         • Internal SBC & Ethernet
                                    • “Insight Manager” on PC’s
                Compute Nodes       • Dedicated systems
                                    • Cluster/node
                                       monitoring & control
                                        • RMS database
                                        • Ethernet &
                                        • Serial Link


                                                                                                     




                                                                    T T       G
                                                                 P IT T S BU RG H
                                                                               PUT N
                                                                 S U P ERC O M P U TIIN G

16                                                               C    E N T E R
     Terascale Computing System
                                             Interactive Nodes
      Control                     Quadrics

     LAN                                     • Dedicated: 2*ES45
                                             • +8 on compute nodes
                                                 • Shared function nodes
                         Compute Nodes
           Interactive                       • User access
            /usr
                                             • Gigabit Ethernet to WAN
 WAN/LAN
                                             • Quadrics connected
                                             • /usr & indexed store
                                                 (ISMS)


                                                                                                          




                                                                         T T       G
                                                                      P IT T S BU RG H
                                                                                    PUT N
                                                                      S U P ERC O M P U TIIN G

17                                                                    C    E N T E R
     Terascale Computing System
                                                      File Servers
       Control                        Quadrics
                                                 • 64, on compute nodes
     LAN
                                                 • 0.47 TB/server [30 TB]
                                                 • ~500 MB/s [~32 GB/s]
                                                      • Temporary user storage
           Interactive
                         Compute Nodes                • Direct IO
            /usr              /tmp
                                                 • /tmp
                           File Servers          • [Each server has
     WAN/LAN
                                                      • 24 disks on
                                                      • 8 SCSI chains on
                                                      • 4 controllers
                                                 • sustain full drive   bw.]

                                                                                                                   




                                                                                  T T       G
                                                                               P IT T S BU RG H
                                                                                             PUT N
                                                                               S U P ERC O M P U TIIN G

18                                                                             C    E N T E R
     Terascale Computing System
                                                         Summary
                                                  • 750+ ES45 Compute Nodes
        Control                        Quadrics   • 3000 EV68 CPU’s @ 1 GHz
                                                  • 6 Tf
      LAN
                                                  • 3. TB memory
                                                  • 41 TB node disk, ~90GB/s
                                                  • Multi-rail fat-tree network
                          Compute Nodes           • Redundant monitor/ctrl
            Interactive                           • WAN/LAN accessible
             /usr              /tmp
                                                  • File servers:
                            File Servers                30TB, ~32 GB/s
     WAN/LAN
                                                  • Buffer disk store, ~150 TB
                                                  • Parallel visualization
                                                  • Mass store, ~1 TB/hr, > 1 PB
                                                                                                                




                                                  • ETF coupled (hetero)
                                                                               T T       G
                                                                            P IT T S BU RG H
                                                                                          PUT N
                                                                            S U P ERC O M P U TIIN G

19                                                                          C    E N T E R
     Terascale Computing System
                                                      Visualization
                                                         • Intel/Linux
         TCS                                                  • Newest software
                        340 GB/s (1520Q)
                                                         • ~16 nodes
                                                         • Parallel rendering
                       Quadrics
                                                         • HW/SW compositing
                                                              •Quadrics connected
3.6 GB/s (16Q)       3.6 GB/s (16Q)          4.5 GB/s (20Q)
                                                         • Image output

       Application                                            •  Web pages +
        Gateways
                           Viz             Buffer Disk

                                                                                                                    




                                      WAN coupled
                                                                                   T T       G
                                                                                P IT T S BU RG H
                                                                                              PUT N
                                                                                S U P ERC O M P U TIIN G

20                                                                              C    E N T E R
     Terascale Computing System
                                                Buffer Disk & HSM
                                                        Quadrics coupled (~225
                                                         MB/s/link)
         TCS
                        340 GB/s (1520Q)                Intermediate between TCS
                                                         & HSM
                                                        Independently managed.
                       Quadrics                         Private transport from
                                                         TCS.

3.6 GB/s (16Q)       3.6 GB/s (16Q)          4.5 GB/s (20Q)
                                                                >360 MB/s to tape

       Application                                               HSM - LSCi
        Gateways
                           Viz             Buffer Disk

                                                                Archive   disk
                                                                                                                     




                                               WAN/LAN & SDSC

                                                                                    T T       G
                                                                                 P IT T S BU RG H
                                                                                               PUT N
                                                                                 S U P ERC O M P U TIIN G

21                                                                               C    E N T E R
     Terascale Computing System
                                                 Application Gateways
                                                          Quadrics coupled
         TCS                                               (~225 MB/s/link)
                        340 GB/s (1520Q)
                                                            •    Coupled to ETF
                                                                 backbone by GigE
                       Quadrics                             •    30 Gb/s

3.6 GB/s (16Q)       3.6 GB/s (16Q)             4.5 GB/s (20Q)


       Application
        Gateways
                           Viz             Buffer Disk

                                                                                                                  




     Multi GigE to ETF Backbone @     30 Gb/s
                                                                                 T T       G
                                                                              P IT T S BU RG H
                                                                                            PUT N
                                                                              S U P ERC O M P U TIIN G

22                                                                            C    E N T E R
     The Front Row




                                                                                     




                                                    T T       G
                                                 P IT T S BU RG H


23   Yes, those are Pittsburgh sports’ colors.                 PUT N
                                                 S U P ERC O M P U TIIN G
                                                 C    E N T E R

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:3
posted:9/14/2012
language:Latin
pages:23