Linux-based virtualization for HPC clusters by nni49141

VIEWS: 15 PAGES: 33

									  Linux-based virtualization for HPC clusters

                         Lucas Nussbaum, Fabienne Anhalt,
                         Olivier Mornard, Jean-Patrick Gelas
                               lucas.nussbaum@inria.fr


                Laboratoire de l’Informatique et du Parallélisme
                   INRIA - UCB Lyon 1 - ENS Lyon - CNRS




Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   1 / 27
 Introduction


Virtualization :

       Subject of a lot of attention in the recent years

       Solves many problems in classic datacenters

       But limited adoption in High Performance Computing




Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   2 / 27
 Virtualization in HPC

Pros :

       Dynamic allocation of resources to job
       e.g automatically scale down a job from 100 to 10 machines

       Easier to share resources between different jobs
       Mix I/O-intensive with CPU-intensive jobs ⇒ maximize usage
       Easy checkpointing of jobs
       Solves a long-term problem in HPC
       Deployment of job-specific work environment
       User no longer limited by administrator’s choices



Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   3 / 27
 Virtualization in HPC

Cons :

       Overhead caused by the virtualization layer
       Poorly understood, rarely evaluated

       Non-exclusive access to hardware

               Performance may vary
               Harder to make use of HPC-specific devices
               High Performance networks (Infiniband, Myrinet)




Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   4 / 27
 Context of this work : HIPCAL Project

       Research project funded by ANR (= French NSF)
       Build virtualized infrastructures
       = virtual clusters spread over the Grid
       Combine system virtualization and network virtualization
          multiple virtual nodes per physical node
           multiple virtual overlay networks on a shared long
           distance communication infrastructure
                Opportunity to allocate bandwidth per user
                (performance guarantees)
       Software : HIPerNET (still in development)
                                           http ://hipcal.lri.fr/
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   5 / 27
 Goal of this work



       Compare and evaluate two virtualization solutions

               Xen

               KVM

       In the context of High Performance Computing




Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   6 / 27
 Xen


       Virtual Machine hypervisor
       First public release : 2003
       Usually uses Linux as dom0
       Developed outside Linux by XenSource (now Citrix)
            DomU support in vanilla kernel
            Dom0 merge under discussion
            Distributions forward-port the Xen patch
       Several distributions have stopped supporting Xen recently,
       and focused on KVM



Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   7 / 27
 KVM


       Virtualization infrastructure integrated in Linux
       = Linux as an hypervisor
       Requires hardware virtualization support in CPU
       First release in 2007 (2.6.20)
       Developed by Qumranet (acquired by Red Hat)
       Relies on QEMU for I/O
           Originally only QEMU emulated devices
           Now paravirtualization with virtio (net, disk)



Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   8 / 27
 KVM : I/O path

                                                                                            Apps

                                                                    Linux
                  Qemu I/O                                          guest                  Linux
                                                                                                      (1)

                                                     (2)
                (3)                                                          KVM module
                              Linux kernel

                         Hardware with virtualization acceleration




Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters     9 / 27
 Evaluation : Benchmarks

Micro-benchmarks :
       CPU
       Disk
       Network
HPC Challenge benchmarks :
   Covers various aspects of High Performance Computing
   Used for the HPC Challenge @ SuperComputing
   Includes Linpack/HPL (used by Top500)
   Other benchmarks included : PTRANS, STREAM, LBB, ...
             http://icl.cs.utk.edu/hpcc/

Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   10 / 27
 Evaluation : Setup

       32-nodes cluster
       Dell PowerEdge 1950 (2 Dual-core Xen 5148 LV ; 8 GB RAM)
       Same userspace, based on Debian sid
       Different kernels (host and guest)
       4 configurations evaluated :
            Xen with full (hardware) virtualization (Xen FV)
            Xen with paravirtualization (Xen PV)
            Standard KVM, using Qemu devices (KVM FV)
            KVM with virtio-based paravirtualization (KVM PV)
       Software versions :
            Xen 3.3.1 + Linux 2.6.18 (from XenSource)
            Linux 2.6.29 + KVM 84
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   11 / 27
 CPU



       CPU-intensive synthetic benchmark

       Using 4 VCPU (mapped on 4 real CPU)

       Almost no memory used

       Compared with time on host system (2.6.29)




Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   12 / 27
 CPU
                                                            102
                    Performance diff. with KVM Host (%)

                                                           101.5



                                                            101



                                                           100.5



                                                            100
                                                                   Host (Xen) KVM   Xen FV Xen PV

                                                          Similar results : almost no overhead
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   12 / 27
 Disk



       Large file written using dd

       Varying block sizes

       Confirmed with bonnie++

       Host system using RAID-0 (expected 120 MB/s)




Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   13 / 27
 Disk

                                                         Host (2.6.29)
                    Disk write speed (MB/s)   200             Xen FV
                                                              Xen PV
                                                             KVM FV
                                              150            KVM PV
                                                          KVM-85 PV

                                              100


                                              50


                                               0
                                                    64        512        4k       32k         256k
                                                                Write block size (B)
            Xen acks writes before they are committed to disk ?
                      Bad KVM+virtio performance
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   13 / 27
 Network


       Throughput measured using iperf

       KVM FV : e1000 NIC emulation

       Xen FV ; Realtek 8139 (no GbE NIC available)

       3 configurations evaluated :
           Communication between 2 VMs on the same machine
           Communication between a VM and a remote system
           Scalability (several VMs, several remote systems)



Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   14 / 27
 Network : inter-VM communications




                                              FV    PV
                                         KVM 648.3 813.2
                                         Xen 96.05 4451



Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   15 / 27
 Network : sending/receiving to remote host
                          sending                                                              receiving

                                              VM                                                  VM

                                                         physical machine   physical machine



                                               1000
                                                             Sending
                                                   900      Receiving
                                                   800
                        Throughput (Mbit/s)




                                                   700
                                                   600
                                                   500
                                                   400
                                                   300
                                                   200
                                                   100
                                                    0
                                                         KVM FV Xen FV KVM PV Xen PV

Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters    16 / 27
 Network : several VM




Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   17 / 27
 Network : several VM - sending
                                   1300
                                   1200                                          1 VM
                                                                                2 VMs
                                   1100                                         4 VMs
                                   1000                                         8 VMs
             Throughput (Mbit/s)




                                    900
                                    800
                                    700
                                    600
                                    500
                                    400
                                    300
                                    200
                                    100
                                      0
                                          KVM FV Xen FV KVM PV Xen PV
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   18 / 27
 Network : several VM - receiving
                                   1300
                                   1200                                          1 VM
                                                                                2 VMs
                                   1100                                         4 VMs
                                   1000                                         8 VMs
             Throughput (Mbit/s)




                                    900
                                    800
                                    700
                                    600
                                    500
                                    400
                                    300
                                    200
                                    100
                                      0
                                          KVM FV Xen FV KVM PV Xen PV
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   19 / 27
 Micro-benchmarks - summary


       CPU : similar results
       Disk : Xen better (but cheats ?), KVM+virtio has problems
       Network :
           Xen PV very good
           KVM+virtio OK
           standard KVM a bit slower
           Xen FV extremely slow




Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   20 / 27
 HPCC benchmarks
       7 benchmarks used by the HPC Challenge
           3 used in this presentation
       Goal : evaluate all aspects of clusters
       8 different configurations :
            32 host system - 2.6.29 + KVM
            32 dom0 - 2.6.18
            32 KVM VM with 4 CPU each, using virtio
            128 KVM VM with 1 CPU each, using virtio
            32 paravirtualized Xen VM, 4 CPU each
            128 paravirtualized Xen VM, 1 CPU each
            32 Xen VM with full virtualization, 4 CPU each
            128 Xen VM with full virtualization, 1 CPU each

Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   21 / 27
 HPCC benchmarks




Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   22 / 27
 PTRANS benchmark




       Sends large messages between nodes
       Measures the total communication capacity
⇒ Result : aggregate bandwidth




Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   23 / 27
 PTRANS benchmark


                           Aggregate bandwidth (GB/s)
                                                         3
                                                        2.5
                                                         2
                                                        1.5
                                                         1
                                                        0.5
                                                         0
                                                              Host (KVM)

                                                                           Host (Xen)

                                                                                        KVM (1V/4C)

                                                                                                      KVM (4V/1C)

                                                                                                                    Xen PV (1V/4C)

                                                                                                                                     Xen PV (4V/1C)

                                                                                                                                                      Xen FV (1V/4C)

                                                                                                                                                                       Xen FV (4V/1C)
            4 KVM VM per node ⇒ better spread the I/O load ?
                       Poor Xen performance
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters                                                                                 23 / 27
 Latency and Bandwidth benchmark




       Messages are being sent in a ring (ping-pong)
       Two sizes : 8 B and 2-MB
       Results : average node-to-node latency and bandwidth




Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   24 / 27
 Latency and Bandwidth benchmark
                                       300
                                       250
                        Latency (us)
                                       200
                                       150
                                       100
                                       50
                                        0
                                             Host (KVM)

                                                          Host (Xen)

                                                                       KVM (1V/4C)

                                                                                     KVM (4V/1C)

                                                                                                   Xen PV (1V/4C)

                                                                                                                    Xen PV (4V/1C)
                               Latency : higher overhead with KVM
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters                              24 / 27
 Latency and Bandwidth benchmark
                                           25

                        Bandwidth (MB/s)   20

                                           15

                                           10

                                           5

                                           0
                                                Host (KVM)

                                                             Host (Xen)

                                                                          KVM (1V/4C)

                                                                                        KVM (4V/1C)

                                                                                                      Xen PV (1V/4C)

                                                                                                                       Xen PV (4V/1C)

                                                                                                                                        Xen FV (1V/4C)

                                                                                                                                                         Xen FV (4V/1C)
         Bandwidth : better bandwidth with KVM (4 VM / node)
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters                                                                   24 / 27
 Linpack/HPL



       Combines computation and communication
          Gives a global picture
       Used for Top500
       Sensitive to latency




Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   25 / 27
 Linpack/HPL
                           250
   Performance (GFlop/s)




                           200

                           150                                                                                                                              Host : 223 Gflop/s
                           100                                                                                                                              KVM 1V/N : 154 (-31%)
                           50
                                                                                                                                                            KVM 4V/N : 152 (-32%)
                            0
                                                                                                                                                            Xen 1V/N : 235 (+5%)
                                 Host (KVM)

                                               Host (Xen)

                                                            KVM (1V/4C)

                                                                          KVM (4V/1C)

                                                                                        Xen PV (1V/4C)

                                                                                                         Xen PV (4V/1C)

                                                                                                                          Xen FV (1V/4C)

                                                                                                                                           Xen FV (4V/1C)
                                                                                                                                                            Xen 4V/N : 183 (-18%)




                                               Xen outperforms all other solutions
                                              Even faster than host with 1 VM / node
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters                                                                             25 / 27
 Conclusion
       KVM and Xen : different performance characteristics
          Xen+paravirtualization generally faster
       KVM : still has some rough edges
       Xen : poor support from distribution, harder to setup
       Future work :

               Other aspects of virtualization : fairness,
               checkpoint/migration
               PCI pass-through for high performance networks
               Re-evaluate with new releases : Xen 3.4, KVM-88
               Linux Containers : promising alternative ?
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   26 / 27
  Linux-based virtualization for HPC clusters

                         Lucas Nussbaum, Fabienne Anhalt,
                         Olivier Mornard, Jean-Patrick Gelas
                               lucas.nussbaum@inria.fr


                Laboratoire de l’Informatique et du Parallélisme
                   INRIA - UCB Lyon 1 - ENS Lyon - CNRS




Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters   27 / 27

								
To top