Building An Enterprise Infrastructure Without An Enterprise Budget A

16/07/2007 Building An Enterprise Infrastructure Without An Enterprise Budget: A Case Study On Virtualisation. Mike Sloane Abstract At the Oxford University Language Centre, we have harnessed the power of Virtualisation to provide a robust, scalable, and cost-effective IT infrastructure that punches far above our economic weight. Through this technology, we have been able to implement unprecedented flexibility, reliability and responsiveness in our IT resources. It has also allowed us to introduce other services that we would not otherwise have been able to fund. This paper introduces virtualisation, available products, the possible approaches, and describes how we leveraged the capabilities of virtualisation to build an infrastructure that is more capable than we could have achieved by traditional methods. The benefits we have experienced are so extensive that I am now planning to implement a Virtual Desktop solution based around VMware ACE for our teaching needs. Introducing Virtualisation Virtualisation in simple terms, is the complete separation of hardware from the Operating System and the functions that it serves. Instead, an intermediary layer is placed between them to manage resources, which permits more than one OS to share the hardware. Each of these Operating Systems is unaware of the others, and believes that it is hosted on its own hardware. Although a new technology in terms of the attention and publicity it is gaining, it is by no means unproven. Many large corporations have adopted virtualisation on a large scale, including BT, Nationwide, Canadian Space Agency, Raytheon, BP, Qualcomm, Merrill Lynch, Siemens and Subaru, among others. There are two approaches to virtualisation, hosted systems, and hypervisors. The hosted approach uses virtualisation software that sits on top of a standard Operating System, and include Microsoft Virtual Server 2005, Parallels Desktop and the VMware Server and Workstation products. This method provides the greatest hardware support, as the drivers used are already incorporated in the host OS. However, there is a performance overhead, due to the many layers and requirement for a fully featured OS between the virtualisation layer and the hardware. This is certainly a viable option for entry-level, low-performance, disaster recovery or test environments, and should certainly be considered for trials if you do not feel confident enough to immediately virtualise your infrastructure. The hypervisor approach replaces the host Operating System with a layer of software that is installed directly onto the x86 architecture, and is sometimes referred to as a “bare metal” virtualisation architecture. Several hypervisor products exist, including VMware ESX Server and XenSource Xen Enterprise. The necessity of managing hardware means that there is still some performance overhead, but it is minimised, and as technology develops, this will be further reduced. 1 16/07/2007 Server Application Guest OS Virtual Hardware Virtualisation Layer Host Operating System Hardware Figure 1: Virtualisation Layers in a Hosted Architecture Figure 2: Virtualisation Layers in a Hypervisor Architecture It has been widely recognised within the IT industry that the hypervisor approach is the way forward, and this can be clearly seen by the inclusion of a hypervisor architecture within the new Operating Systems being introduced to the market, including, Windows Server 2008, Red Hat Enterprise Linux (RHEL) 5, Novell SUSE Linux Enterprise Server (SLES) 10 and Sun Solaris 10. The XenSource products themselves are closely related to the new offerings, as they are all based on the open-source Xen hypervisor architecture. Currently, VMware ESX Server (also encapsulated within Virtual Infrastructure 3), and XenSource XenEnterprise are the leading hypervisor products, and have a massive head start in the development and adoption of their products. Microsoft’s new hypervisor, code name Viridian will be an integral part of Windows Server 2008, however, it will not be available until May 2008. Microsoft has also announced that it is scaling back some of the features that will be included in this initial release[17]. Hardware efficient virtualisation is difficult to achieve due to the limitations of the x86 architecture, which was not designed with this application in mind. To maximise hypervisor efficiency, there are three differing approaches to virtualisation[5]. Transparent virtualisation allows self-contained virtual machines to run without modification to their operating system. This is the most OS-independent of the approaches. Secondly, there is Paravirtualisation, which uses a modified Operating System within its virtual machines. It is claimed that this method has performance advantages; however, this is questionable in current hypervisor implementations. The final method is Hardware assisted virtualisation, in which the server hardware itself supports virtualisation. Hardware support for virtualisation is currently in its infancy and products to provide this are rare, however, both Intel and AMD have released processors with onboard virtualisation support, and many manufacturers have devised roadmaps to provide solutions for virtualisation. In the long term, as virtualisation becomes a mainstream, the three approaches will provide complementary technologies that will enable enhanced performance. Now to answer the question of what advantages virtualisation has to offer. We are all under pressure to provide more powerful systems, more services and more reliability with less funding. There is also greater demand for more flexibility and responsiveness from IT systems. These are all things that it is extremely difficult to achieve in a traditional environment; virtualisation is the answer. The impact of virtual infrastructures on enterprise IT and the power that it gives to IT staff is enormous. 2 16/07/2007 With virtualisation, we can achieve the following: • • • • • • • • • • • • • • • • Server consolidation Extend the life of legacy applications and retire old equipment Scalable services More efficient use of resources Reduce IT expenditure Centralise management and simplify administration Allocate resources as required Delivery of 24/7 business continuity Greater reliability Simplify and speed up disaster recovery Rapid server provisioning Highly redundant infrastructure Reduced power, space and air conditioning requirements Reduced expenditure on hardware and maintenance Testing environment included More efficient use of IT staff time As mentioned earlier, there are a number of products available for virtualisation; which of these you choose depends upon what you want to achieve with your virtual infrastructure. If you want a testing environment, you could use VMware Workstation, Parallels Desktop or Microsoft Virtual PC. The capabilities of Microsoft Virtual PC are very limited; it doesn’t provide the flexibility that could have proven useful and merely acts as a console to the virtual machine. It is less fully-featured than VMware Workstation; it provides the capability to create and run virtual machines, and it does it well, but that is all it does. The GUI is clunky, and all VM configuration is done through a Wizard, which can be time consuming and frustrating. OS support is also very limited, focuses on Microsoft products, and only offers 32-bit support. In addition to this, it does not provide the capability to build VMs with multiple virtual processors, or build a fully functional virtual network. You can network your VMs, but there isn’t the ability to see what’s going on or build a completely isolated virtual network for testing purposes, as there is no built-in virtual switch capability. When you initially configure the Virtual Machine, your configuration options are very limited, so it is difficult, if not impossible, to replicate a system that may be representative of your real-life situation. This limits the software’s accuracy, and therefore usefulness as a test bed. My favoured testing environment is VMware Workstation. I have used it extensively, and found this to be an easy to use and fully featured tool. I could create virtual machines (using a large range of guest Operating Systems), and with Virtual Machines that accurately represented their hardware counterparts. I found that Workstation had many useful tools and functions that let me experiment and test things to destruction, and go right back to where I started. I could provision VMs from templates, and reduce resource use by using Linked Clones. Instead of providing a full VM, Linked Clones use the virtual disk of the machine it was cloned from. Not only does this save resources, it also makes provisioning VMs much quicker. Moreover, there is no impact on what you can do with the clone; it is essentially an independent machine. I was also able to build, observe, and manipulate networks of virtual machines to model and refine my designs for a new domain infrastructure. I could see how the components interacted and what the structure looked like. You can see this both from the VM level, or an aerial view, which is extremely useful when you want to optimise and manipulate the network to 3 16/07/2007 achieve a specific goal. If you want to, you can link your network with the real world, or with self-contained virtual networks on other machines, so you really do get a flexible and fully-functional design and test environment. The general feeling with this product is that you have much more control, and can get much more information from it without losing any usability. It is essentially a tool that provides a much more accurate representation of real life systems and infrastructures. If you just want to consolidate your servers onto fewer machines, there are two products you could use: Microsoft Virtual Server 2005 or VMware Server. Each of these runs on top of a host Operating System, so there is a tax on resources, but the virtualisation software is free. This is a good way of creating a virtual testing environment, or consolidating servers onto a single box if you have no budget or don’t need to guarantee resilience. If you are new to virtualisation, and just want to try it out to see what all the fuss is about, I recommend that you download one (or more) of these products and give it a go. I don’t intend to discuss these products any further here as they are entry level products that do not offer the capabilities and advantages I aim to highlight. It should however be noted that Microsoft Virtual Server 2005 is based on Microsoft Virtual PC so subject to most of the same limitation mentioned above (Windows OS only, no SMP). The final options available are the Enterprise level products built on a hypervisor architecture. These currently include VMware Infrastructure 3 (using ESX Server) and XenSource XenEnterprise, but Microsoft Windows 2008, RHEL5, Novell SLES10 are all reported to include this capability[8]. At their most basic, the products are available to allow only server consolidation; however, some of the products are more fully featured, and there are add on products that can extend this capability massively. For the purposes of this paper, I shall only be considering VMWare VI3 and XenSource, which are currently the only proven hypervisor products on the market. XenEnterprise is based upon the open-source Xen hypervisor architecture, and has adopted the Paravirtualisation approach of using modified Operating Systems for virtual machines that it hosts. Xen claims that this is a more efficient use of hardware resources, as the guest operating systems can access them directly (rather than by using binary translation), however, Xen’s own benchmark data[8] shows that the only real benefit comes when using more than 2 processor sockets on your server (note: sockets, not cores). There would either be a massive skills requirement to paravirtualise an OS, a time delay while Xen does it for you, or you would be restricted to one of the OSs they have pre-prepared. Currently the selection of preprepared Operating Systems is very limited. A paravirtualised infrastructure may also be unsuitable in a mixed physical/virtual infrastructure because paravirtualised Operating Systems cannot run on “bare-metal”, which would create a requirement for multiple versions of each operating system in use. However, recent advances in Hardware virtualisation technology (Intel-VT and AMD-V) now allow XenSource to run unmodified guest operating systems[8]. Performance of the current generation of hardware assist technology is only average and does not offer any significant benefit. This situation is likely to change in the future with the introduction of second generation hardware assist technologies, such as Intel NPT (Nested Page Tables). VMware ESX Server uses its own hypervisor architecture, and uses the transparent virtualisation approach. ESX uses a combination of Binary Translation and Direct Execution to achieve as near to native speed execution as possible. In physical server implementations, the Operating Systems’ kernel communicates directly with the hardware it is hosted on (using privileged kernel calls). In a virtualised system, the guest operating system instead has to communicate with the hypervisor, which manages the hardware resources. This arises from the need to divide physical 4 16/07/2007 resources, allocate them to each VM, and manage them on behalf of each VM. Transparent virtualisation means that instead of modifying the Operating Systems used on the virtual machines to allow this, a binary translation layer is introduced to convert privileged kernel calls (ring 0 data) coming from the virtual machines into less-privileged application layer code that the hypervisor can then use to communicate with the hardware. The virtual machines are effectively running as applications on the hypervisor. Where possible, such as in the case of applications hosted by VMs (which do not require privileged access to hardware), Virtual Machine code is run directly on the hardware – this is called Direct Execution. VMware offers four Virtual Infrastructure packages to meet your requirements and budget. The entry level product is VMware Starter Edition, which will allow you to consolidate a number of servers onto one physical box, but doesn’t include any of the added-value features that make VMware stand out from the competition; although there is an upgrade path to allow you to add these capabilities at a later date. Starter Edition only offers the option to store VMs locally or on a NAS appliance; there is no support for shared storage such as Fibre Channel or iSCSI. If you wish to add live VM migration (VMotion), high availability and load balancing, you will need to purchase an Upgrade license to VI3 Standard or Enterprise Edition. Another drawback is that the Starter Edition is limited to 4 CPUs (potentially not a problem), but more seriously, it’s also limited to 8GB of Physical Memory[7]. You could get this level of functionality, albeit not so efficiently, by using VMware Server, or by using the XenExpress. Your choice between these two products will be determined by which OS you want to use (whether it is supported), and whether you might want to migrate to a VMware product later. However that said ESX claims to return twice the number of Virtual Machines of VMware Server so if you intend to run many virtual servers it may be worth the license cost. The Standard Edition, is really the starting point for any virtual infrastructure that will offer a long lifespan. This edition provides ESX Server, and a fully functional file system that does support clustering, and also supports Virtual SMP. Again, many of the features that make VMware products stand out from the competition are not included, however, these are offered as add-on packages, and you can pick and choose which capabilities you would like to add to your system as your need and budget allows. You can in effect build a fully functional VI3 Enterprise Edition package in instalments, by adding VMotion, high-availability, load-balancing and backup capabilities, allowing scalability within your datacentre. Enterprise Edition comes with all of these features included. However, it does not include the Virtual Centre software that you need to take advantage of these additional features. All of the additional components that are included with VI3 Enterprise are agents that plug into Virtual Centre, which then manages the cluster automatically according to preset rules (or rules specified by you). It is a shame that this is an additional expense, and Virtual Centre does not come cheap, but it does increase the benefits of having a virtual infrastructure exponentially and can manage up to 128 ESX servers. A final package exists for to allow easy transition from a physical server infrastructure to a Virtual Infrastructure. VMware Infrastructure Acceleration Kit gives you Enterprise Edition Licenses for 8 CPUs, Virtual Centre, Gold Support, and VMware Converter, which allows you to convert your physical servers into Virtual Machines. This package is aimed at providing organisations that already have a large data centre the opportunity to gain the benefits of a virtual infrastructure without undertaking a disruptive and costly redevelopment programme. Formerly known as P2V Starter Edition, and only available to purchase, VMware Converter is now available on the VMware website as a free download; and is 5 16/07/2007 therefore available to anyone using VMware products. It provides the ability to convert a number of physical servers to virtual machines simultaneously and without downtime, and create virtual machines from other VM or Backup formats[16]. ESX can either be managed at the command line, as any other Linux system (it should be noted that ESX is not based on Linux; only the special management console used to interact with ESX is), or by using the included Virtual Infrastructure Client. The GUI itself is very powerful and easy to use, so there should be very little need to use the command line. This interface is identical to the Virtual Centre interface, but does not include the cluster management functionality, and only allows you to manage a single server at a time. Within this interface, you can view data on configuration, resource allocation, and performance at either the server level, or VM level. You can add VMs and resource pools, edit Virtual Machines to add or remove virtual hardware, take snapshots of VMs, and perform all the administrative tasks that you would expect of a physical server. You can also use the console view function to gain direct access of your virtual machines as you might use Remote Desktop or VNC (although it is much more responsive than VNC). The client software runs on your local machine; you can download it by navigating your web browser to the ESX host. VMware has built their virtualisation platform with security in mind from the beginning. This has been sufficiently proven for a number of government and defence organisations, and financial institutions to adopt VMware in their production environments. They have received NSA and Common Criteria Evaluation and Validation Scheme security validation [13][14]. ESX server is completely isolated, and has no communication with the outside world except through the Service Console. The VMkernel’s design limits it to supporting Virtual Machines, and it runs only those services that are essential to perform this specific function. Insecure and nonessential services such as Telnet and FTP are not installed, and the ports are closed. The default install of ESX is highly secure, with all outbound ports closed, and only inbound ports necessary for the virtual infrastructure are open. Because the hypervisor uses binary translation to modify communication between the guest OS and the hardware, the guest OS runs in a less privileged state, which restricts its capabilities and therefore ensures the security of the host and other virtual machines. The ESX service console is protected by a firewall, and communication between ESX Server and the VI client/Virtual Centre are secured using SSL (256-bit AES block encryption and 1024-bit RSA key encryption)[6]. Figure 1: Isolated ESX VMKernel ensures security VMware Virtual Machines are completely isolated from one another. This means that should one VM crash or be breached, it will have no effect on the other VMs running on that host. Furthermore, an administrator for one VM or resource pool cannot gain 6 16/07/2007 access to another VM or resource pool unless specifically given privileges to do so by the ESX Systems Administrator. Each VM is only aware of the resources that have been specifically allocated to it. Those VMs or Administrators of those VMs will not even be aware of any other virtual machines on the host. Resource pools allow cluster resources to be divided and delegated, enabling group-level administrators to provision and manage virtual machines, but prevent them from exceeding their quota or accessing more resources than they are entitled[5]. Resource pool boundaries are strict and isolated, so resource usage in one pool does not affect the available resources in another pool, and the machines and administrator of one pool cannot see any of the other resource pools that may be present on the cluster. VM isolation extends beyond Administration however, and just as in a physical network, Virtual Machines can only communicate with other Virtual Machines running on the same host through the use of a virtual switch. The same applies to data stored on iSCSI or Fibre Channel, where the Virtual Machines only see the Virtual Disks that are assigned to them, working in the same way as LUN masking. Each Virtual Disk is owned by a single VM, and while that VM is running, locks prevent access to the Virtual Disk by any other ESX Host. No two VMs have access to the same virtual disk unless shares are configured within the VM, or the Virtual Machines are configured for clustering as with Microsoft Cluster Services and the like. An abstraction layer between the Virtual Machine and the hardware maintains all memory maps, and means that a VM can always write to, and only access its own memory space. There have been no known incidents of VM "break out" (i.e. someone breaking out of one VM into another VM, or the ESX system itself) since ESXs release in 1999. Virtual machines have the same vulnerabilities as their physical counterparts. The easiest way to attack a virtual machine continues to be poorly configured or unpatched guest operating systems. Providing you use the same security measures within a Virtual Machine that you would use within a physical server (firewall, antivirus, etc), it will be no less secure than a physical server. ESX server automatically divides available physical resources equally among the VMs, which ensures protection against DoS attacks. This isolates the physical resources of the VM, so that if one is compromised the others remain in service. This is not a prescribed allocation however. You can customise these settings according to Server requirements by changing the resource reservations and limits for each individual VM. Background to Virtualisation at the Language Centre Language Centre IT has historically suffered from a lack of investment due to inadequate access to financial resources. These restrictions have lead to difficult operational choices being made on whether to invest in IT infrastructure, or run more classes to meet the ever-growing demand. There had been sporadic IT investment when additional funds could be found, but much of the existing infrastructure had been implemented in an ad-hoc manner, and the things that could not be achieved at low cost, were not implemented. The result was an infrastructure that had been stretched beyond its capacity, and mainly consisted of obsolete budget-brand equipment. The situation was unsustainable, the entire infrastructure, including workstations, network and server had become unusable; its inherent unreliability caused significant downtime, shattered user confidence in IT, and created a substantial maintenance overhead. Furthermore, there was no possibility of adding new capabilities, or even scaling up to accommodate more workstations. After analysing our existing IT situation to look for areas for improvement, it became clear that there was no easy solution to these problems; the whole infrastructure had 7 16/07/2007 exceeded its useful life, and an improvement in service could only be achieved by addressing the network, server, and workstation problems as a whole. Figure 2: Old Network Architecture The first phase was to design and implement a new network infrastructure, and then replace the obsolete workstations. A decision to install a gigabit network with Cat6 cable arose from three considerations; first, I anticipate a growth in internal bandwidth requirement as our use of IP based multimedia grows; second, as we had an opportunity to start from scratch, it was a sensible time to make the upgrade; and finally, since we’re in an old building and laying cables is difficult, I wanted to ensure the network has a long lifespan. This allowed us to move on to the second stage, which was the installation of a new digital classroom that would allow us to integrate IT and multimedia facilities to provide a powerful tool for language teaching. With this completed, and a significant improvement already noted, it was time to address the more complex issue of servers, and services that we would provide. This was made increasingly urgent because the hardware was no longer supported by either the manufacturer or the maintenance company, and we had already used up the components cannibalized from similar retired machines. Furthermore, the legacy Netware operating system was no longer supported by Novell. We were living on borrowed time, and needed to replace the legacy system we were using with a scalable, reliable, fault-tolerant solution that would potentially have a long lifespan. I also wanted to add many more capabilities to improve the user experience, and address the short-comings of the existing system. Although it was recognised that there was no alternative to making the investment, we had to do it as inexpensively as possible, while ensuring that the project achieved the best quality and results possible. 8 16/07/2007 From the outset of this project, I decided upon some very specific and realistic criteria that it had to achieve: • • • • • • • • • • • • • • • • Increase reliability, flexibility and availability Improve performance Better use of resources Limit and localise problems Improved security Increased responsiveness to new requirements Enhance business continuity and disaster recovery capabilities Provide scalable infrastructure Improve ability to meet future business needs Maximise infrastructure lifespan Reduce administrative overhead Improve cost-effectiveness of IT operations Provide an infrastructure which is easy to use and manage Minimise capital expenditure Minimise ongoing costs and power consumption Take into account limited floor space, power supply and air conditioning After auditing the current system, assessing current technologies, likely trends and future requirements, it seemed like there were two routes we could take. Firstly, we could install a similar system to the one we already had, i.e. a single server hosting all our required applications, including file-server, authentication, and distributing applications. Other than providing more powerful hardware, a more capable operating system and manufacturer support, this option would have given us no real benefits over the existing system, and retained all its weaknesses. The second option was to use the traditional method of building a solution with hardware, and following the recognised best-practise of hosting just one or two services per server. Although reluctant to go down this route because of the costs, complexity, and building works involved, it looked much more favourable than the first option. Adopting this method would have allowed fault-tolerance through duplication, however, to do it properly would prove expensive, wasteful, and would require a drawn-out implementation, as we didn’t have the available budget to get the whole system operational. We could meet all our business and design requirements, however, it would also introduce a number of other issues to contend with, such as space requirements, air-conditioning and power provision, and would have introduced the additional complexity and unreliability of clustering. In March 2006, Alex Mittell from OUCS introduced me to a third possibility; Virtualisation. His recommendation was specifically a VMware solution; he did however, make me aware of offerings from Xen and Microsoft. My first experience of a Virtual environment came very shortly after this conversation, when I attended an instructor-lead training course. The training provider used VMware Workstation to allow us to build fully functional virtual machines and virtual networks without having to constantly rebuild and reinstall. It proved itself to be a very powerful, flexible and efficient tool, and impressed me greatly. After returning from the course, I continued to use VMware Workstation for my test environment, which had previously been a multi-boot workstation. Although efficient with hardware, the multi-boot setup meant that I had to continually rebuild the machine when something went wrong with one of the Operating Systems; Virtualisation eliminated that drawback completely. This success, lead me to ask whether Virtualisation would allow me to achieve all the criteria I had set out for the replacement server infrastructure. 9 16/07/2007 The initial problem I faced was that I knew very little about virtualisation, the available products, their hardware requirements or capabilities. I also had no way of judging my own requirements as I had no systems to use as benchmarks. Finding benchmark data for VMware installations proved difficult, and I later learned that the End User License Agreement (EULA), prohibited this data from being published without VMware’s consent. It was also a concern that VMware themselves didn’t publish any benchmark data. Although the technology promised a great deal, I had concerns that its capabilities at enterprise level had been over-hyped as with so many other innovations. I had a number of concerns about hardware performance, security and encapsulation of VMs, and whether the technology was proven. There were some case studies available, and some documentation, however, these didn’t provide the data that would allow a good judgement to be made. Initially, I found that there were plenty of documents to show how virtualisation worked, and testimonials to say that it did work, but nothing more specific. I contacted several server manufacturers, explained what I wanted to achieve, and asked them for their recommendations. It quickly became apparent that they themselves weren’t too familiar with the practicalities of virtualisation, and either had no advice to give, or over-specified their systems massively. The most clear example of this was the HP reseller I met with, who went away to get his engineer’s advice, and came back saying that I would need a server that matched VMware’s maximum specs. Since I was only planning to run 6 Windows VMs, I found this to be very unlikely. Other manufacturers similarly over-specified their systems, but were much closer to what I would eventually implement. The most useful insights were gained by attending the annual VMware customer seminar in London. As well as meeting and talking with engineers from VMware, and various server manufacturers and suppliers, keynote presentations were given by customers who had already implemented a virtual infrastructure on a large scale. They gave quantitative details of their needs, considerations and achievements, their successes and difficulties. They also gave demonstrations showing that their new infrastructure actually worked. Furthermore, the keynote speakers were from large corporations including BT and the Nationwide Building Society which gave me the confidence that this technology had been proven in a mission critical environment. The information provided by the speaker from BT particularly demonstrated the value of Virtualisation. He had virtualised all their servers, and reduced the scale of their infrastructure from 5 data-centres to three. He also showed one of their active ESX nodes, which was a single-core, single-processor machine, hosting 26 Virtual Servers. At last, I felt that I had a starting point for hardware specification, and I could clearly see the cost savings. Language Centre Architecture At the time of designing the new virtual infrastructure, there were some major limitations with the Xen products. The range of supported operating systems was very limited (actually, only two Linux distributions), iSCSI was not supported (only Fibre Channel), there was no support for virtual-SMP, and the management console was less flexible. Xen also lacked many of the added-value features of ESX, including hardware failover capability, automated load-balancing and resource scheduling. There was also some debate over whether Windows and Linux Virtual Machines could be run on the same physical box; a restriction that is not present in any of the VMware products. Rather than offering a solution for a fully functional virtual infrastructure, it provided a very good tool for those who just wanted to achieve server consolidation. In our situation, I aimed to use the power of virtualisation to maximise performance and resilience of our infrastructure at the lowest possible cost. The VMware product range provided all these features, and 10 16/07/2007 was available immediately; in short, VMware is still the technological lead in virtualisation, and Xen has a lot of catching up to do. Using VMware Virtual Infrastructure 3 as the foundation, I was able to design an IT infrastructure that would meet all of the design goals, and provide an avenue for continued growth and future support. One of the key facilitators of this is the complete independence of both hardware and virtual machines, which will allow the virtual machines to be replaced as new Operating Systems are adopted, and hardware to be upgraded or replaced, all without interruption to service; enabling a true rolling replacement programme. Virtualisation has offered us so much flexibility that I am confident we will be able to meet all of our future needs within our virtual infrastructure. The starting point for the design was to consider the services that I wanted to provide, and what OS to use: • • • • • • Infrastructure Services: Directory Service, Authentication, Group Policy, Folder Redirection, DHCP, DNS, WINS Patching and Antivirus: WSUS, Sophos EM Library Print and File Services Applications Server to host Melissi Language Lab software Altiris Deployment Solution for imaging workstations Intranet and Internet Web Servers We had traditionally used Novell Netware for our server, but since this was going to be a completely new infrastructure, it was worth considering all of the options available to us and making a judgement based on which most closely matched our needs. Following an analysis of our needs, and some research, I decided to adopt Windows Server 2003 infrastructure; it was clearly the best tool for the job. It most closely and simply matched our business needs, and offered a very powerful tool to enable us to reach our goals. Other factors that lead me to this conclusion was that a) our workstations were all Windows hosts, and b) we use many beta-quality language teaching applications that were only built to run on Windows machines. Past experience has shown that even with considerable and costly re-engineering, they do not function well on other Operating Systems. We reaped the benefits of this decision when Microsoft later announced that it is permitted to run 4 Virtual Machines per license if you purchased Windows Server 2003 Enterprise Edition. Since we required a number of Windows VMs, we got additional functionality for the equivalent cost of buying individual Standard Edition licenses. I rejected Novell for a number of reasons. It was a class-leading operating system in its day, but a number of poor decisions meant that it lost its position. Netware has become a niche OS; its competitors have caught up with and in some areas surpassed its capabilities, and I believed it to be a dying OS. I also had concerns about the company’s lack of direction (at the time), and therefore the ongoing quality of products and support. The company did not appear to be able to provide enough resources for the development of Netware or SLES concurrently, and didn’t seem to know which it wanted to focus on. I had also worked in Netware environments previously, and have never been completely happy with any Netware implementation I had come across; this opinion was strengthened when I discussed the possibilities with people who were implementing the latest incarnations of Netware, and found themselves running into considerable difficulties. 11 16/07/2007 Linux was not appropriate for our environment; one of the things I was keen to introduce was out-of-the-box standardisation and simplification, which due to the infinite possibility of Linux customisation was unachievable. I wanted to allow other administrators to be able to come in and quickly see how the infrastructure works without having to having to rebuild it from scratch. Although documentation would help this, out-of-the-box standardisation would speed this process up. Since we were exclusively using Windows clients, it made more sense to have a homogeneous infrastructure than invest the additional time and research required to set up a Linux infrastructure. For example, additional time would be required on configuration, and on deciding which of the many applications would be the most appropriate to use to deliver a particular service. Linux however, is the best tool for our Web Servers, and I currently have two Red Hat Enterprise Linux AS 4 Virtual Machines. I decided to use Red Hat Linux as that is the distribution that we were already using, and we had 3 years remaining on our support contract. I would have been equally happy to use SUSE, or looked at any other distribution that someone happened to recommend. System Design In its simplest form, our system consists of: 2 Single Processor (Dual Core) rackmount servers connected to 2 3.5TB RAID 5 iSCSI SANs via a Managed Gigabit switch. This is all managed by VMWare Virtual Centre. We are currently operating at a 5:1 consolidation ratio, but have the built-in capacity to go to approximately 30:1 including spare resources to allow failover (more physical memory would be required). If for some reason, both cluster nodes fail, the VMs can be re-deployed by installing VMware Server or Player on Workstations while the main system is brought back online. Our infrastructure was made possible by the terms of the VMware sales model, which licenses the software in per-processor (socket, not core) form rather than perserver. This meant that I could use one 2-CPU VI3 license to build a cluster that provides fault tolerance. There has been some debate over whether this is legal, however, it seems to be acceptable under the terms of the EULA, and discussions with VMware Partners, and VMware directly indicate that it is acceptable, but unsupported. Discussions about this on the VMware Forum also draw this conclusion. It has been very difficult to find concrete data on this issue, which does not appear to be advertised on VMware’s Website, however, within their White Paper “VMware Infrastructure 3: Pricing, Packaging and Licensing Overview” (2006), it is made clear both within the Summary, and under the heading “Capacity Pricing” that this is legal. VMware state “ processor capacity purchased can be deployed or redeployed on any mix of servers as required”[7]. The cost of an additional license would have been prohibitive, and the additional processing capabilities provided by two more CPUs would have been wasted on our needs, so this licensing model is our enabling factor. There is no doubt that the VMware products are much more expensive than competitor products, however, the added-value you get in terms of functionality, support, and power is unsurpassed. The two servers provide a VMware cluster to provide redundancy in the event of hardware failure. All data, including the Virtual Machines, is stored on the iSCSI SAN, which allows the VMs to restart if one of the cluster nodes fails. Communication between the Servers and the SAN is handled by a software iSCSI initiator on each server, and there is no problem with performance. While both Servers are running, VMWare Virtual Centre (which is itself hosted on a Virtual Machine), monitors and automatically load balances the VMs between the two cluster nodes. When new VMs are created, Virtual Centre will decide which host to allocate it to according to the resources available. Additionally, rules can be created that will keep VMs on the 12 16/07/2007 same cluster node if required, or on separate cluster nodes (as is the case with our Infrastructure Servers). For security, the SANs have been configured to only accept data transfer communications from the two cluster nodes, using IP Deny/Allow rules. This can be changed using either the web interface, or the console, if required. The browser-accessible GUI interface is very easy to use, and makes configuration of the SANs very flexible, and very simple. Our two iSCSI SANs are both configured to RAID 5, formatted for VMFS (which is essential for VMware clustering), and have been divided into 2 LUNs. The first LUN on each SAN is active, and linked to the VMware cluster. The second LUN on each SAN is configured for real-time block level asynchronous replication of the first LUN of the other SAN. This provides a number of benefits. Rather than have one active SAN, and one backup SAN, we can double the effective bandwidth by using all 4 of the teamed NICs to transfer data to and from the cluster. The data is also spread among more disks on different systems, so data access is also speeded up. Replication traffic also travels on this path, but the impact of this does not have a detrimental effect on performance in our environment. Hardware redundancy is built into the SANs, but if one SAN fails, all the data is available on the second SAN, and a 5 minute manual intervention can point the cluster at the second LUN of the remaining SAN to make all the data available. It may be possible to automate this process with a script; however, to ensure there is no loss of data, I have decided to retain this manual step. The result is that we effectively have RAID 5+1 with added hardware redundancy. The motivation behind this design was to ensure maximum uptime, and maximum utilisation of the system’s capabilities. I was also aware that we were moving from an environment where all staff kept their data on a local machine to a scenario where we had all-our-eggs-in-one-basket, and a SAN failure would have resulted in disruption and possible data loss for many more people. In addition to this cross-replication, each of the Virtual Machines is backed up to the centralised TSM system hosted at OUCS. The main weaknesses of the TSM system are the link between the Centre and OUCS, and the length of time it takes to do a restore. It is very useful as an offsite backup, and for file-level restore operations. For additional data security, a snapshot of each VM is taken regularly, enabling an instant restoration of service should one of them become irreversibly corrupted, and all user data is replicated using the Volume Shadow Copy services built into Windows Server. Figure 3: SAN Cross-Replication There is some supplementary hardware that is required to make this system work, namely one or two gigabit switches to allow communication between the cluster components, and one or two UPS devices. To ensure total redundancy and failover, it is best to have 2 switches and 2 UPS appliances. In the Language Centre, we have: 13 16/07/2007 1 x HP ProCurve 2810 24 Port Managed Gigabit Switch 2 x APC Smart UPS 3000VA Uninterruptible Power Supplies Each server and iSCSI appliance has two PSUs built in, and these have been alternately connected to the two UPS appliances to ensure that the system stays online should one of the UPS devices fail. Figure 4: New Network Architecture I have created two virtual machine templates from which I can quickly provision any number of standardised VMs. These can then be modified to meet the specific requirements of the role they will serve. Windows Server 2003 Enterprise Edition VM • Single Processor • 20GB System Disk (HDD) • 512MB RAM • Floppy Drive • CD Drive • Single NIC Red Hat Enterprise Linux AS 4 VM • Single Processor • 10GB System Disk (HDD) • 384MB RAM • Floppy Drive • CD Drive • Single NIC I have specified the HDDs included in the template to be used only as System Disks. This means that a relatively small disk can be included in the template, and the 14 16/07/2007 System can be separated from Data. This enables maximum flexibility as additional disks of appropriate size can be added to store the data for the provisioned server’s specific function. It is a quick and simple matter to add another Virtual HDD to the VM through the Edit Settings function within the VI Client. Servers After researching the products of several manufacturers, including Dell, IBM, HP and Fujitsu Siemens, I decided to purchase the servers from Dell. The products of each of the manufacturers was satisfactory; build quality was good, but none offered anything that substantially differentiated them from the others. Dell however, was best in two areas, firstly, the price was much better, and secondly, Dell have designed their systems with some forethought. They had clearly considered that their products may be in-service for some time, and developed a roadmap for providing commonality between their servers to ensure their support. They had also designed their servers with future upgrading in mind, and clearly mapped, among other things, an upgrade path for the server’s components, including the processor. The potential to extend the life of the hardware was one that was particularly valuable to us. Server Hardware Specification: Chassis: Processor: Memory HDD RAID Controller RAID Level Optical Drive Redundant PSU NIC Riser Cards Dell PE1950 1U Rack Mount Intel Xeon 5160 3GHz, 4MB Cache, 1333FSB 4GB 667MHz (1GB Dual Rank DIMM) 2 x 146GB 3.5” SAS, 10,000RPM PERC 5/I, 256MB Cache 1 CD/DVD RW Combo Yes Intel PRO 1000PT Dual Port Server PCIe Broadcom 2 Port, TOE (not enabled) PCIe By clustering these servers, the resources are pooled, providing 8GB RAM and 11GHz of processing power to the virtual machines that they host. If the need arises, we can improve server performance by: • • • • • • • Adding more Memory (up to 32GB per server) Adding a hardware iSCSI Host Bus Adaptor Upgrading the Processor (higher speed or more cores) Adding a second Processor and purchasing a second VI3 License Upgrading the hard disks to faster drives Enabling TOE PCIe bus ensures that higher speed NICs can be installed in future Having implemented the system and seen it running for several months now, I would have done two things differently. I would sacrifice processor speed, which is overpowered for my environment, and increased the amount of memory instead. Current average processor usage levels are below 1.5GHz (of a possible 11GHz), with demand increasing to 6GHz at peak levels, such as when a scheduled Sophos “extensive" scan on all 9 servers is initiated simultaneously. Memory usage however 15 16/07/2007 hovers between 45% and 65% during normal load. Although sufficient for my current needs, I intend to keep the system at or below 60% utilisation to allow sufficient capacity for all of the systems to failover if necessary. I found it interesting to note the different levels of resource usage depending on whether the guest OS is Windows 2003, or RHEL 4. The Windows Servers use between 25-40MHz of CPU and 350-500MB of Host Memory (40-75MB Guest Memory). The RHEL Web Servers use between 110-120MHz of CPU and 350-360MB of Host Memory (7-25MB Guest Memory). iSCSI SAN I did not want to entertain using Fibre Channel due to the high costs involved, and the fact that this hardware was overkill for our environment, so I looked at the other options available. iSCSI was the best of these options, and although it is also easy to spend a lot of money on iSCSI devices, it is possible to find good quality solutions at lower prices. After ruling out a number of suppliers on price, it came down to two, FortuNAS, and PowerServers. FortuNAS systems appear on the VMware Hardware Compatibility List (HCL), as long as a hardware iSCSI HBA is used. This does not mean that it is required, but if you use a software iSCSI initiator, VMware support may decide not to help with related issues. The PowerServers products are not yet on the VMware HCL, and so are unsupported by VMware. Both products are comparable in terms of quality and features; both use Intel chassis, and Open-E operating system. The only difference seems to be that the OS used by FortuNAS offers NAS capability where the PowerServers product is only iSCSI (Open-E iSCSIR3 Enterprise). I obtained quotes from both suppliers, and could get a single 2TB chassis from FortuNAS for £5,112, whereas PowerServers could offer 3.5TB for £2,550. After comparing specifications, and enquiring about the feasibility of using the PowerServers system with ESX, I decided to purchase their equipment. Since I had budgeted for the FortuNAS system, I could get 2 PowerServer iSCSI SANs, which would allow me to achieve one of my other long-term aims of having on-site redundancy for data storage. When I received the SANs, I was surprised at their high-quality, the extensive functionality of their Operating System, and the ease of configuration. I was also surprised by the flexibility of the hardware, which would allow you to partition the disk array enabling you to have different RAID levels for different parts of the same box, and you could also add more iSCSI boxes to form a single array. I could write an essay on the features it offers, but this information is freely available. I would however say that I highly recommend it; its build quality is excellent, and it offers functionality found on much more expensive high-end kit. After connecting them to the Servers, I found performance to be very satisfactory, although large and intensive database applications may prove a strain. SAN Hardware Specification: Chassis Processor Memory NIC HDD RAID Level OS Redundant PSU Potential Points of Failure Power iSCSI 4008 Intel Xeon 3040 Dual-Core 1.86GHz, 1066MHz FSB 1GB DDR2 Dual Gigabit 8 x 500GB SATA II 5 Open-E iSCSI Enterprise Yes 16 16/07/2007 Due to financial constraints, it has been necessary to accept a number of potential points of failure within this system, however, every effort has been made to minimise the impact that these will have on the users’ daily activities. The first point of failure is the gigabit switch; I would like 2 for redundancy, however, there are other switches in the cabinet, so if it fails, I can take some of the less important workstations off-line and patch in the VMware cluster. The second potential point of failure is power. Unfortunately, it was necessary to put both UPS devices on the same circuit, otherwise major cable re-routing would have been required. This may be a future modification. The final weakness in the Centre’s setup is the single link with the outside world; however, the internal infrastructure was designed with self-sufficiency in mind, so if this link fails, disruption should be limited to email and internet. Current Virtual Server Provision Server 1 Server 2 Server 3 Server 4 Server 5 Server 6 Server 7 Windows 2003 Windows 2003 Windows 2003 Windows 2003 Windows 2003 Windows 2003 Windows 2003 MS SQL Server 2005, VMware Virtual Centre Domain Controller Active Directory, Group Policy, DNS, DHCP (20% Scope), Global Catalogue Domain Controller Active Directory, DNS, DHCP (80% Scope), WINS, Global Catalogue WSUS, Sophos EM Library Application + Print Melissi Digital Language Lab Server Application Server MS SQL Server 2005, Altiris Deployment Solution File Server Redirected User Data: Application Data, My Documents and Desktop folders Web Server PHP/HTML Intranet Pages, Web page for registration of SSO Credentials with local Domain Web Server Project Website Virtual Centre Server Server 8 RHEL 4 AS Server 9 RHEL 4 AS Future Developments in the Centre I already have plans to provision more Virtual Machines in future to provide additional services, these include: • • • • • Exchange Server for shared calendaring and Mailing-lists. Online Course Booking Web server for the Departmental Website Server(s) to host digital audio and video resources Certificate Servers I can provide these services for just the cost of the software, which is the enabling factor, for instance, I can build the PKI by creating virtual machines for the Offline Root Certificate Authority and Enterprise Subordinate CAs. After authorising the Subordinate CAs, I can turn off the Root CA, burn it to disk and put it in the safe; so I have complete security as if I had a computer to dedicate to the Root CA, which I kept switched off in a locked room. By using Windows Server 2003 Enterprise 17 16/07/2007 Edition, I will have the option to use Version 2 certificate templates, which offer more flexibility and configurability. Another problem that we are facing is a growing need to provide more workstations within the Language Centre for student and self-study use. This is becoming more important as the more useful language teaching resources are computer based, including websites, applications, and digital media. The capital outlay for this would be a massive proportion of our Departmental budget, and in truth, would not be a value-for-money investment. These computers will by no means be used intensively, and with the endless progression of technology, their lifespan will be too short to make such an investment really worthwhile. However, the provision of computing facilities to facilitate language learning is recognised as being essential. A solution to this dilemma, which I am currently investigating, is the Virtual Desktop technology that has come from VMware, specifically VMware ACE. This new product will allow me to build secured Virtual Desktops, containing all the applications necessary for language learning, for deployment to student laptops. The Virtual Desktop can be configured to only function within the Language Centre, and an expiry date can also be set to render the VM useless when the student’s membership expires. Furthermore, security issues arising from students using their own laptops can be negated as ACE allows only the Virtual Desktop to function on our LAN. So, although the students provide their own hardware, they would be using our system. This overcomes a number of issues: first, we would not have to worry about the expense of providing and replacing hardware, the students will do that themselves. Secondly, we don’t have to worry about creating images for the machines as new software comes out, we can just add the software to the VM and deploy it. Thirdly, we don’t have to worry about the security of student machines, because we will be able to dictate the environment in a form that could not be overridden. I am still at the early stages of researching this product, but I have yet to find a fault with the solution that it provides. A Virtual Future For Oxford University I can see adoption of virtualisation technology rapidly escalating in Oxford, the question however, is whether people will adopt the same product. Some of the reasoning behind choices will be matching product capabilities to requirements, others will be budgets, but I hope that choices will not be made on the basis of software ideology. I can see a huge benefit to the IT infrastructure if we choose to work together. I would like to see this opportunity used to create an Oxford-wide VMware Grid; a highly-redundant infrastructure where a number of Departments and Colleges can collaborate to host backup Virtual Machines for each other to provide warm-site availability. This would mean that if the server room in one department became unavailable for some reason, the servers would be waiting for activation on another VMware cluster within the University. Those Departments that have a need and can afford to implement a Virtual Infrastructure could also provide unmanaged hosting for smaller Departments by using resource pools, allowing everyone access to the very best technology. By embracing virtualisation, this could easily be possible, since the expense traditionally associated with server duplication will be avoided. Conclusion Virtualisation has gained industry-wide acceptance amongst hardware and software vendors, and considerable effort is being put into the development and refinement of this technology. There is sufficient evidence to support the viability, reliability and security of this technology and it is clearly recognised as the way forward for infrastructure provision. This powerful and innovative solution can provide 18 16/07/2007 unprecedented flexibility, redundancy and cost-saving, while eliminating a degree of complexity, cost and administrative burden associated with providing a highly available data centre. Tools have been provided by the software vendors that enable easy and non-disruptive conversion from physical to virtual machines, making the switchover process relatively simple and painless, and many large corporations and government agencies have already adopted the technology. Virtual Infrastructure provision is a rapidly developing field, with OS manufacturers beginning to incorporate virtualisation technology into their products and several virtualisation specialists producing products to meet that requirement. VMware’s products still lead the field by a substantial margin, and look set to maintain this lead for at least the next few years. Although its products look like a substantial up-front investment, the scalability and cost savings can be clearly illustrated, and a virtual infrastructure of varying capability can be achieved on even very small budgets. Vendors have introduced a clear roadmap to allow smaller adopters to add additional capability to their virtual infrastructure as additional funding becomes available to them. It is undoubtedly becoming a mainstream technology and its adoption within almost every organisation is inevitable. In addition to the benefits outlined in this paper, virtualisation, it also provides environmental benefits. So far, the IT industry has remained mostly unscathed in the global warming argument, but we do undoubtedly use a lot of power, and virtualisation will allow us to cut that power requirement and reduce our carbon footprint (I am running 6 appliances instead of 18). In most cases, a failure to incorporate virtualisation into current plans for IT improvement programmes shows a very short sighted and wasteful approach that will result in outof-the-box obsolescence. There is no downside. Acknowledgements I would like to acknowledge the contributions of Alex Mittell, whose VMware expertise and general support and advice were essential to this venture, and for reviewing and amending this document. Also Adrian Parks, who developed the system to register University Single-Sign-On accounts with the Language Centre domain, and who along with Nick Barber, gave other more general advice and assistance when needed. Jon Hutchings was also a great help in reviewing this document. These guys are all highly skilled and experienced, and I highly recommend them to you for your own projects. Contact Mike Sloane IT Officer Oxford University Language Centre 12 Woodstock Road Oxford OX2 6HT Email: michael.sloane@lang.ox.ac.uk References: 1. Windows Server Virtualisation – An Overview, Microsoft Corporation, (May 2006) 2. Virtual Server 2005 R2 SP1 Product Overview, Microsoft Technet, (June 2007) 3. Virtualisation Overview, VMware, (2006) 4. Building the Virtualized Enterprise with VMware Infrastructure, VMware, (2006) 19 16/07/2007 5. Virtualisation: Architectural Considerations And Other Evaluation Criteria, VMware, (2005) 6. Security Design of the VMware Infrastructure 3 Architecture, VMWare, (2007) 7. VMware Infrastructure 3: Pricing, Packaging and Licensing Overview, VMWare (2006) 8. A Performance Comparison of Commercial Hypervisors, Xen Source, (2007) 9. XenSource Product Overview: www.xensource.com 10. VMware Product Overview: www.vmware.com 11. VMware Technology Network www.vmtn.net 12. VMware Forum http://www.vmware.com/community/index.jspa 13. VMware NSA Security Certification http://www.vmware.com/company/news/releases/nsa_pr.html 14. Common Criteria Evaluation and Validation Scheme review of ESX Server http://niap.bahialab.com/cc-scheme/st/?vid=10056 15. http://www.vmware.com/company/news/releases/paravirtualization.html 16. VMware Converter http://www.vmware.com/products/converter/ 17. Technet Magazine UK Edition (July 2007) 20 16/07/2007 Table 1: Summary of Features VMWare VI3 OS Support Unmodified Linux Microsoft Windows Novell Netware Solaris x86 FreeBSD Paravirtualised Guest OS Added Value Intelligent and automatic load-balancing across cluster Automatic Failover Ability to Move Live VMs between Hardware Templating Snapshots On-The-Fly Guest Customisation Hot Add Disks Removable Device Support Encapsulated Disks Yes (1999) Yes (1999) Yes (2002) Yes (2006) Yes Announced (2005) XenSource XenEnterprise Yes (with Intel/AMD VT) (2006) Yes (with Intel/AMD VT) (2006) No No No Yes (2004) Yes Yes Yes Yes Yes Yes Yes Excellent Yes No No Not Easily No No No No Poor Supported But Not Recommended (Uses LVM Volume per VM) Yes Yes Hardware Support for Virtualisation (Intel VT/AMD-V) Yes (64-bit) Yes (32-bit, when more efficient than BT/DE) 21 16/07/2007 Table 2: Cost Comparison – Hardware vs. Virtualisation Virtual Server Implementation Component Server 1 Server 2 Storage 1 Storage 2 UPS (x2) Windows 2003 Licenses VMware VI3 VMware Virtual Centre Total: Traditional Server Implementation Component Server 1 Server 2 Server 3 Server 4 Server 5 Server 6 Server 7 Server 8 Storage 1 Storage 2 Backup for Server 3 Backup for Server 4 Backup for Server 5 Backup for Server 6 Backup for Server 7 Backup for Server 8 UPS (x 4) Windows 2003 Licenses Total: Function Domain Controller Backup Domain Controller File Server Security Server Altiris Server Melissi and Print Server Web Server Internal Web Server iSCSI Storage iSCSI Storage Redundancy Redundancy Redundancy Redundancy Redundancy Redundancy Cost 2066.00 2066.00 2066.00 2066.00 2066.00 2066.00 2576.00 2576.00 2560.00 2560.00 2066.00 2066.00 2066.00 2066.00 2576.00 2576.00 3076.00 Function Cluster Node 1 Cluster Node 2 iSCSI Storage iSCSI Storage Cost 2,416.44 2,347.36 2,560.00 2,560.00 1,538.00 564.56 1,952.00 1,500.00 15,438.36 2822.80 41,982.80 Figures have been normalised for a base installation, and do not include costs of software hosted by the systems. Both installations require 2 instances of MS SQL Server, but for different purposes, and so this cost has been excluded. The figures assume that identical failover and redundancy capabilities are to be achieved whether the system is implemented in hardware or by virtualisation. Also note that there is no capacity for expansion built into the traditional (hardware) server implementation model; this is inherent in Virtualised environments. 22