Docstoc

Virtualization+Vmware

Document Sample
Virtualization+Vmware Powered By Docstoc
					The Grid and Virtualization
                     Orran Krieger
        Sr. Technical Staff Member
                            Vmware
                                           What is Virtualization?
• A level of indirection between hardware and software
• Virtual Machines: a full abstraction of a physical machine
    •   Safely and efficiently multiplex virtual hardware on physical hardware




            Virtual
            machine


            Physical
            machine


• Benefits: resource sharing, portability, compatibility



                                       2
                                          Server Virtualization




Before Server Virtualization:
 Single OS image per machine
 Software and hardware tightly coupled
 Running multiple applications on
  same machine often creates conflict
 Underutilized resources
 Fault/security vulnerability


                                      3
                                             Server Virtualization




Before Server Virtualization:             After Server Virtualization:
 Single OS image per machine              Virtual machines (VMs) break 1-to-1
                                            dependency between OS and HW
 Software and hardware tightly coupled
                                           Manage OS and application as single
 Running multiple applications on
                                            unit by encapsulating them into VMs
  same machine often creates conflict
                                           Hardware-independent: they can be
 Underutilized resources
                                            provisioned anywhere
 Fault/security vulnerability
                                           Strong isolation between VMs


                                      4
           VM Encapsulation and Mobility


                                    Suspend
 Log
                                    Resume
                                    Snapshot
   VMM                VMM
                                    Clone
                                    Migrate
                                    Record
                                    Replay
                                    etc.

Virtual Machines are more versatile than Physical Machines




                            5
       Traditional ESX Server
                                           98%                                                          2%

          Agent      …   Agent       RPM




                   RHEL3-based
                  Service Console
                                                    Helpers               VMM          VMM      VMM

                                                                                             VMkernel
                                                     Storage                    Networking

                                                               Resource Management

                                                              HAL and Device Drivers




Disk Footprint:                  2 GB
                                                 Disk Footprint:                 32 MB
Percent of Patches                >50%



                                           6
                                 ESX Server 3i: Thin Virtualization!
                                                        98%                                                        2%

    Small footprint…
           Agent
                    withAgent     RPM
                         OS-independence
       Virtualization should have the reliability,
       security, and performance of hardware


    Less code and fewer interfaces:
       Compact 32MB footprint
       CIM- and remote CLI-based management
                   RHEL3-based
       Aids both security and reliability (no legacy)
                  Service Console
                                                                 Helpers
    Integrated in server systems:                                                      VMM        VMM      VMM

       Hardware is certified and ready-to-run                                                           VMkernel
       Simple start up experience                                 Storage                    Networking

    Launched with IBM, HP, Dell, FSC, NEC.                                  Resource Management

                                                                           HAL and Device Drivers
    In the next few years, virtualization will
    be pervasive embedded in all HW
Disk Footprint:                2 GB
                                                              Disk Footprint:                 32 MB
Percent of Patches              >50%



                                                        7
  Grid and grid-like technologies are the future
• Web applications being developed with scale-out
  frameworks
• MapReduce and related frameworks…, Data Intensive
  Supercomputing
• Utilities inside companies
• Hosting companies like Rackspace
• Shared utilities like Amazon’s EC2
• The rise of SAAS like salesforce.com
• HPC capacity and capability systems


  Grid and Virtualization are complementary:
   Are both technologies really going to be pervasive?
  •Grid: motivated by analogy of power grid
  •Virtualization: converts computation into a fungible
  commodity; hugely simplifies realization of grid

                             8
                  Traditional Grid Adoption Challenges

• Security
   • Grid jobs can potentially compromise security on host machine
   • Intruders may piggy-back on Grid jobs to attack the system
• Isolation
   • Grid jobs may “eat” into host machine's resources
   • Buggy grid job may crash host system
   • OSes often can’t control resource usage between Grid jobs
      • Forces single job per host models
• OS heterogeneity limits number of Grid nodes available for a job
• Service level guarantees
   • Enterprise grid jobs need guarantees on completion time

• In many cases, applications must be re-written for Grid




                                            9
Traditional Grid Adoption Challenges (Contd)
• Provisioning
   • Increased activity on host system can slow down Grid jobs
• Utilization
   • Dedicated machines for Grid jobs lead to severe under-utilization
   • Legacy applications fail to utilize multi-core features
• Management
   • Grids lead to increase in number of “nodes”
   • Management of these nodes is a big challenge
• Infrastructure for application virtualization
   • Grid middleware lack support from infrastructure layer for on-demand
     deployment of application instances
• Grid implementation limited by having to use commodity OSes
   • Large attack surface, high overhead, …




                                          10
                    Virtualization in Grid Computing
• “Any problem in computer science can be solved
  with another level of indirection” (David Wheeler)
• Virtualization provides a natural level of indirection
  to address Grid Computing challenges
  • Inherent “cures” to Grid adoption issues
  • Same solution applies to broader utility, hosting, enterprise,
    web … marketplace




                                11
                     Benefits of Virtualization in Grids
• Virtual machines (VM) provide robust security
  • A compromised VM can be isolated without affecting the host
• VMs provide excellent isolation
  • A Virtual Grid node (VGN) crash has no effect on other VGNs or host
• VGNs can guarantee service level times
  • VMs can be provisioned with required explicit resource reservation
     • CPU bandwidth, memory, disk space, disk bandwidth, etc
  • Ensures the grid job has the muscle power to complete within the SLA
• Virtualization reduces data centre complexity
  • Abstracts out infrastructure heterogeneity




                                     12
       Benefits of Virtualization in Grids (Contd)
• No resource usage creep with VMs
  • Provisioning places limit on resources that can be consumed
  • Very useful for “resource scavenging” Grids
     • Host performance does not deteriorate

• Multiple VGNs on a single physical machine
  • Increases utilization of Grid boxes
• Enhances workload management capability of the Grid
  • Additional VGNs can be started and deployed instantly
• On-demand provisioning of VGNs eliminates OS heterogeneity
  issues
  • As many VGNs of a particular OS as needed




                                               13
       Benefits of Virtualization in Grids (Contd)
• Dynamic resource management
  • (1) Vary resource allocated to a VGN based on load/requirement
  • (2) Move VGN to a more/less powerful host
• Provisioning additional grid nodes becomes trivial
  • Maintain repository of pre-created VM images
  • Choose and bring-up VGN based on requirement
• Ease of management
  • Infrastructure can be managed through virtual machine management
    solutions
  • Maintenance and upgrade of VMs easier than physical servers
• Custom OS support
  • Special purpose OSes, specialized for Grid applications can be
    developed, resulting in major performance, reliability, simplicity
    advantages.
  • Lets the Grid application get much closer to the metal.




                                      14
                    Grid Computing

                         Grid Job

                                            Grid Job



         Grid Job




                              Grid                     Grid Job
                          Middleware




                        Grid Job          Grid Job



Jobs IN/OUT


                                     15
                    Virtualization based Grid
Jobs IN/OUT             Grid Middleware




         Grid job          Other Job




• Grid middleware can focus on “core” Grid
  issues.
• Virtual infrastructure provides simplified abstraction
  used by Grid middleware and other applications.


                              16
                 VMware’s GRID enablement features

• Base features: low-overhead, strong isolation…
• Integrated management of virtual and physical
  infrastructure.
• Guaranteed quality of service:
  • VMotion to move tasks between physical machines
  • Distributed resource scheduling
• Reliability:
  • Snapshotting for checkpointing of long running jobs
  • High availability
• Cheap creating of VMs
  • memory overcommit
  • templates & clones
• Special purpose appliances



                                 17
                     Towards Distributed Virtualization




                                             RP1A    RP 1B

                                               RESOURCE           RESOURCE
                                                 POOL               POOL

 OS        OS




Physical Machines   Machine Virtualization          Distributed Virtualization




                                      18
                                                                 VMotion™
• Migrate running VMs from one server to another
   • Encapsulation and HW independence are the key enablers
• To minimize service disruption:
   • Iteratively pre-copy VM memory from source to destination
   • Store virtual disks on networked storage: FC, iSCSI, NAS
• VGNs moved across ESX hosts with zero downtime; No need to restart Grid
  jobs
• Allows efficient usage of hardware resources and improves overall hardware
  utilization
• Similar VGNs with can be grouped together for better performance




                                              19
               Storage VMotion

     • Storage independent migration of
       virtual machine disks
       • Zero downtime to virtual machines
       • LUN independent
       • Supported for Fibre channel SANs




20
  Automatic load balancing across hosts

Distributed Resource          Dynamic Balancing
 Scheduling (DRS)           Continuous Optimization




                       21
                Adding and removing hosts

                    Add/remove capacity on demand
Hot-plug machines    Distributed power optimization
                     Improve application availability



    DRS




                           X
                    22
     Transforming capacity management

PHYSICAL   VIRTUALIZED                          POOLED




                         R                  R
                         P                  P
                         1                  2



                                  HW   HW       HW        HW   HW
                                                                    HW
                                  HW   HW       HW        HW   HW

                                  HW   HW
                                                     HW        HW   HW
                                  HW   HW




               Logical Resource Pooling (RP)
               Distributed Resource Scheduler (DRS)


                             23
                       What is a Virtual Appliance


• Pre-built, pre-configured and
  ready-to-run software
  application packaged with the
  OS inside a Virtual Machine.




• Or packaged inside multiple
  Virtual Machines




                                  24
          Virtual Appliance OS


•Appliance OSes can be
simplified or customized
• Don’t need complex
  hardware management
• Don’t need broad application support
• Highly-customizable, application-
  specific OS
      • Include only what application needs.
• Look at hardware appliance operating
  systems for examples




 25
                                                Concluding remarks
• Virtualization and Grid will both be pervasive
  technologies
  • Virtualization will increasingly be embedded on all server platforms.
  • Various forms of utility computing are increasingly dominating many markets.
• Virtualization and Grid are complementary.
• Its time to start working together:
  • What is the right interface between Grid middleware and virtual infrastructure?
  • What are the requirements of the Grid? Determinism for collective operations? Fast
    start up of VM? Fast Clone…
• Its time to write Grid appliances/OSes.
  • How tiny can they be? Plan9?
  • Does a Library OS model make sense? Libra/Prose?
  • Is there any paravirtualization support that would be useful?

Orran Krieger <okrieger@vmware.com>




                                        26
                                           Thanks


• Hariprasad Nellitheertha
• Chinmay Albal
• Radhika Kulkarni
• Larry Rudolph; Steve Strassman; Scott Davis;
  Jack Lo




                         27
28
                          VMware Memory Overcommit

• Unique VMware Feature
• Host system memory is usually the resource
  bottleneck
• VMware memory management technology allows an
  average of 2:1 memory overcommit (and oftentimes
  greater)




       Transparent Page Sharing        Memory ballooning




                                  29
                         Key Properties of Virtualization

          Partitioning                                 Isolation




  Run multiple virtual machines              The virtual machine is isolated
  simultaneously on a single PC             from Host OS, other VMs on the
                                                        same PC

        Encapsulation                     HW Independence & Transparency




Entire virtual machine is saved in         Run a virtual machine on any PC
files. Can be moved and copied               without modification. Break
        like any other file.               dependence between OS & HW



                                     30