San implementation and configuration guide TestingMonitoring the SAN

Setting up Datacore SanMelody software alongside Vmware ESX By Paul Winterbourne 24.07.07 Version 1.1 Index Introduction. Installation guide for ESX server and virtual center pc/license server. Network configuration: general config/iscsi and multipathing io San Configuration: initial setup and replication. How to vmotion between the two esx servers. Performance Monitoring Backing up/Restoring virtual machines/virtual center and esx hosts. Other tasks (patching, firewalls, remote into esx etc) Links Page 3 Page 5 Page 8 Page 15 Page 22 Page 25 Page 26 Page 27 Disclaimer: The Author if this work holds no responsibility for your systems of you attempt anything written in this document. Try it at your own risk!. 2 Introduction This document is the culmination of many hours of tinkering, reading, head banging and most of all fun. There is certainly an amazing learning curve when it comes to vmware. On top of this is the learning curve involved with Datacore’s Sanmelody software. Normally people would have consultants in to set up this sort of thing though me being me I decided to do it all myself for the hell of it. I found that there were quite a few grey areas or areas that I had to scour the web for alongside the help of many people in the VMWare forums in order to get the new infrastructure up and running. Hopefully this paper will help others out in areas where I was stuck. The whole paper is based around the idea that you will have two esx servers and two san servers. I am sure there will be those who will wish to expand from this but this is the smallest set up one can have for complete resiliency. What this design will do for you is give you san failover(using iscsi replication and multipathing io) and esx server failover (using VMWare HA). Obviously HA means you have to have your vm’s reboot but hey most smb’s can live with this!. The whole reason for this project for my company was to limit the amount of time it takes to rebuild servers in case of a disaster. Not only that but some of our servers have no documentation from previous software installations so by p2ving them and replicating the vm’s we can escape from the hazard of not knowing how to rebuild a server. The design will use three subnets so you will have to amend your network as you want. The first subnet is for the main network (pc access to the virtual machines etc) the second subnet is for iscsi between esx and the sans (this includes vmotion and alternate pathing etc) and the third subnet is a direct link between the two Datacore sans for replication. I was interested in the idea of doing this to a remote site. It’s hard to figure in all the factors but it could certainly be possible and maybe better than that infamous 16.5 minutes someone else claimed. I would sure like to try it and try and get my head around the design. Anyway you can email me if you have any questions but like all of us I have a job as well so I can’t guarantee I will answer your emails, similarly I am new to all this as well. Oh also I have chopped out a fair bit as I am sure most of you know how to activate the licenses etc. Really this document will focus on the core parts of sanmelody and VMWare and show you what to do in order to get it working. Excuse the way I have written some of it as it was originally written for my company. You’ll get the gist of it though. 3 Glossary:- Network Managed Volume: A virtual volume that is created on the san. You create a NMV in sanmelody and then it can be seen by vmware. In my set up I create nmvs as 2TB so that it always shows as 2tb to vmware esx server. However, it does not necessarily have anywhere near 2tb underneath it (another name for this is thin-provisioned disks). It may, for example, only have 500gb. You can set the san up to give you a warning if you are close to the nmv “real” limit. By setting it up in this way you can add storage on the fly if you don’t have enough space to begin with. I will show you later on how to add storage to an nmv pool. The reason we use nmv’s are twofold. The first is we don’t need to reconfigure vmware esx server when we add more storage and second when setting up mirrors between the two sans replication takes 3-4 days by doing it the other way (i.e. just using virtual volume replication as opposed to network managed volume replication which , by the way, includes the use of virtual volumes as well but puts another layer of abstraction in between NUMA: Non Uniform memory access. This is a way of increasing the speed by using local memory access to/from a processor and can speed up systems that use multiple processors. (See performance tuning section). All you really need to know is that vmware esx is numa aware and that in your servers you MUST have the same amount of memory on both sides of the machine(there are two rows of memory banks within the hp servers so you must have equal memory on each side i.e. if you have four blocks of 2gb on one side then you must have four block of 2gb on the other). I had a few niggles with this (see the performance chapter/numa) 4 Installation guides for esx servers and virtual center. ESX Installation: Basically the set up for esx is pretty straightforward. Insert the esx cd and boot off it. (Make sure you have mirrored the drive using the disk configuration utility built into the bios on the hp’s). When you install the cd choose all the normal options and make sure that you install over any preconfigured system partitions (the ones that usually include eisa partition or tools. Having said that I was not sure if this was the right thing to do, however I did do it this way so only time will tell if this was the wrong choice. It may well affect the mirror). Choose an Ip address for the servers, I chose 128.1.x.x for esx1 and 128.1.x.xfor esx2 (give both machines fully qualified names but less than 30 character i.e. esx1.domain.co.uk, if reinstalling make sure you delete these from the dns records and the active directory before reinstalling if they exist). I made the password “xxxxxx” and the username “xxxx” for both machines for easy access. Once you have installed esx you then need to go and install the virtual center machine. The reason you have to install virtual center is because you are going to install the licensed feature for both esx machines on the virtual center pc. You have to install the esx licenses on the virtual center machine rather than the esx hosts themselves if you want to use vmotion and clustering (VMWare high availability). Virtual Centre installation: Install virtual center using a typical installation, make sure you choose to install msde (you will have to figure out how to use sql server if you go down this route) and make the virtual center pc a local license server as well (this will also ask you to point the pc to the license file which is wherever you have stored it). Tcp ports for virtual centre (leave as default and obviously turn the xp firewall off!!): 5 Make sure you tick the boxes above for apache. Once you have installed virtual center (you will most probably be using msde at this point and you better note that this is only supported in a test configuration not a production server config but hey we are using it as we can’t afford another sql license at the moment) you can then either access the virtual center by going to a web browser and typing in it’s ip address or use the virtual infrastructure client on the virtual center cd which has more features and is better (you need to install it on to your workstation) and then log in (note if going straight to an esx server you have to use the root username and password but virtual centre login uses domain authentication). You then need to put both esx hosts under the control of the virtual center pc. You do this by creating 6 a datacenter in the virtual center admin console (see page 33 of the quick start guide vmware official documentation on how to do this including how to add licenses to both esx hosts in virtual center). Do also note at this point that there is a bug in esx where if you reboot an esx machine you may lose your licensed features under the configuration tab in the virtual center. If you then try to add the licenses back in following page 33 it won’t show that you have any licenses. To resolve this issue reboot the virtual center pc as it will free up the licenses (or restart the virtual centre service); you will then be allowed to add the licenses to the servers again as necessary. You must add the licenses before trying to set up iscsi as it will not allow you otherwise. You will now have to configure the iscsi link between the sans and the esx hosts. In order to do the vmware side of things you need to create a vmkernel link to the virtual switch first. (I will outline this in the network configuration chapter below). Put esx servers under the control of the virtual center: Log into the virtual centre by going into the virtual centre using the VI client. Now click inventory, then hosts and clusters and click new datacenter. Give it a name. Now add a host to the virtual center inventory (i.e. esx1) do make sure you use the hostname (check you have given your vmkernel dns settings correct and use the root username and password). Avoid using the ip address as I have seen problems later on with this. Now add esx2 as a host to the inventory Now click the licensed features section under the configuration tab and tick the esx standard type license and click add ons (add everything). Do this for both esx servers. 7 Network configuration: general config and iscsi The network and iscsi config must be set up first to enable you to set up replication. Sanmelody must be installed on the sans as well. In order to understand how the SAN and vmware will work I have had to utilize everything at my disposal in terms of the current network infrastructure at Lloydmaunder. I think it is far easier to understand from a diagram rather than text. I won’t explain in this section why I have set it up the way I have but lets just say that this diagram took a lot of time to construct and is the answer!. The only thing you basically need to know is that there are three subnets (and this should never change otherwise layer 2 loops will occur) and that if one esx server fails or if the primary san fails there are alternate paths that can be taken in the event of a failure. If you are an administrator who is here after I leave please do not make any changes to this unless you really know what you are doing and what impact you are going to make. I would advise you to keep things the way I have designed them unless you are happy with the setup and make yourself familiar with this diagram!. 172 subnet: a dedicated loop through (at wholesale meat) using fibre between c site and poultry 10 subnet: Fibre link looped through every fibre panel on backbone (only way I could do it!) 128 network: Normal Live network where Lloydmaunder pc’s and domain reside on 8 The first thing to do is set up the network card ip address on the sans. Each san has 2 nics, one nic will be on 10.0.0.0 network and one on the 172.0.0.0 network. The second thing to do is to configure the nics on the esx servers themselves. One nic should all ready have a 128.1.0.0 network attached to it (as you specify this on installation of esx server). So you need to set up the other nic with a 10.0.0.0 address. This can only be done by creating a new virtual switch and assigning it a network card (it is then that you specify the ip address of the network card). See below for this Setting up a VMKernel on both esx servers: You need to set up the virtual networking on both esx servers as follows: esx1: Esx2: 9 So first you need to set up a virtual switch linked to the 128.1.0.0 network card (this vswitch should all ready be there. So just click its properties and add a vmnetwork(this is where the virtual machines link into and allow traffic on this subnet to reach the virtual machines) and a service console to allow you to ssh into it). Now add another vswitch by clicking add networking (give it a label, note that the label is linked to the physical adapters and you link the physical network adapter by adding it to the vswitch once the vswitch is created). Now you can add a vmkernel to the second vswitch which is linked to the 10.0.0.0 network adapter. This allows the vmotion network on the 10.0.0.0 network (vmotion is handled by the vmkernel and therefore the vmkernel needs to be on the 10.0.0.0 switch) (see diagram below) Log into the virtual center management console and click on the esx server you wish to add the vmkernel to. Now go to the configuration tab and then click the network link. Under here click add and then select the vmkernel from the list. Give it an ip address on the 10.0.0.0 network.(I gave them 10.0.xx and 10.0.xx) Set the default gateway as well. This should be done for both esx servers (as they both need it to use iscsi storage). To set up iscsi links to the storage see the iscsi setup section below… 10 (example of adding a vmkernel to a vswitch…please note the above diagram is wrong it should say 10.0.0.6 with a subnet of 255.0.0.0) Microsoft Iscsi Initiator setup (used for the replication between the Datacore sans not anything to do with esx): If you want to have a brief technical overview of iscsi then go here http://en.wikipedia.org/wiki/ISCSI http://tools.ietf.org/html/rfc3720 http://www.dtc.umn.edu/publications/reports/2005_06.pdf Basically all you need to know is that Iscsi works by having an initiator and a target. The Initiator software for us comes in the form of the esx servers(vmware built their own initiators) and the targets are the datacore sans (it is a bit like databases where you have a client and a server i.e. an initiator and a target). However we also need to add Microsoft initiators onto the sans themselves so that replication can occur (this is because you cannot link a target to a target so what you are doing is linking a Microsoft iscsi initiator on san1 to the sanmelody target on san2 and linking the Microsoft iscsi initiator on san2 to the sanmelody target on san1 for failover replication). This is done by downloading Microsoft iscsi initiator to both sans and setting it up to point to each san target (the point here is that you have two network card in each server. One 11 network card acts as a target for the esx servers on a 10.0.0.0 subnet whereas the second network card acts as a target and an initiator for replication (172 subnet) and the subsequent failover replication when needed between the sans).When installing the Microsoft iscsi initiator it is imperative that you do not install the Microsoft iscsi mpio option. When you first start the Microsoft iscsi initiator on the sans absolutely make sure that multipathing is NOT ticked. (otherwise replication will never work between the sanmelody servers). Once you installed the Microsoft iscsi initiator it will appear in the datacore mmc (install a vanilla install of sanmelody at this point). Configuring the iscsi initiator on both machines is very simple. Just make sure that you go to the discovery tab and add in the ip address of the iscsi target (read more below in this section about ip setup). It will then find the iscsi target and add it into the targets tab. (see diag) also make sure that you enable multipathing and enable automatically restore when you initiate your initial connection. At this point make sure that you have configured, in 2003 control panel, the network cards on the san (i.e. one is on the 172 subnet physically connected to the other san and one is on the 10 network connected to the esx host). In the datacore mmc you can add a target (do this on the primary san first…) to the ms initiator by right clicking it and add. The target should be the ip of the 2 nd san on the 172 network see diagram below… 12 You should set up the Microsoft iscsi initiator on san1 to look at the 172 ip address of san2 (i.e. the target on network card 2 on san2). You do this by right clicking on the ms iscsi initiator and adding the target. You should then set up on San2 the ms iscsi initiator to look at 172.x.x.x (san1 nic). It is always a good idea to go into the ms iscsi config in control panel to double check the ms settings (sometimes it may over rule the datacore settings). Make sure that in drop down boxes where it says default you specify the correct ip addresses. Also in the ms iscsi config window you can connect (i.e. logon to the target) so make sure that has been done, also you can specify if you want a persistent connection. YOUR ISCSI CONNECTIONS AND CHANNEL MAPPINGS ARE KEY!!! TAKE AN HOUR TO GET YOUR HEAD AROUND THE SETUP AND HOW THEY WORK. The diagram below shows a rough guide as to how all the iscsi connections hang together in the set up. However the diagram that comes after this is better as it actually shows the real life set up and iscsi hostnames etc (the other diagram is in the next section). I Will not tell you how to setup iscsi in sanmelody and esx yet as you need to setup sanmelody and esx first. Anyway here is a quick diagram (better one later in san and esx iscsi setup section) 13 14 San and ESX server configuration: initial setup and replication. A quick word on San Melody: Always click the apply button after you complete an operation. You can only work in one mmc at a time (as you may flick between sans) so make sure you don’t have an mmc open on the other san. If you want to make some changes on san2 and san 2 is in read only mode you can click the lock in the mmc and it will put it into write mode. Obviously this puts san1 in read mode. Sometimes you need to close the mmc for the screen to refresh. Also when you create a mirror set, first it will show that the mirror is broken and redundancy has failed. This is normal. Just close the mmc and leave it a few minutes and it should come up fairly quickly within a short while. Make sure you set up the disk mirror between the disk arrays (really it is a mirror between an nmv virtual volume and an nmv, note one nmv has not been served up) before mapping virtual volumes to application servers. Another thing you may need to be aware of is when recreating nmvs and mirrors from scratch (i.e. rebuilding the raid arrays from scratch) you may need to reboot both sans (hopefully you will never need to do this!). It is important to set up the configuration in the following way (these are not detailed instructions that comes later and before): 1.) install esx on to servers 2.) make sure network infrastructure is correct excluding iscsi but install ms iscsi on sans (as per the diagram) 3.) Set up network adapters with relative ip addresses and subnets 4.) Vmware: install virtual centre, add datacentre and add hosts and licenses. 5.) create raid sets on disk arrays but do not format them with windows! (I.E. use your raid software for use with the raid controller you have installed). 6.) Install Microsoft iscsi initiators on both sans and configure them to look at each others targets (on the 172.1 network i.e. these ms initiators are for replication between the Sans) 7.) Add licenses and activate sanmelody and datacore if not done so all ready. 8.) Add administrator account to datacore domain services service in services(that is the services under control panel and administrative tools) on both sans (see sanmelody help guide under prerequisites for high availability document. You may need to put your sans on the main network domain for this so you will have to set up a method of routing between the 128 network and the 10 network, I did this by using an old router and configuring it then adding the sans into ad as per usual 2003 server route, do realise though that this router will be essential later on when it comes to logging in to the sans if you login with the domain password). See the instructions below. 9.) Set up the Partnership between the two sans. 10.) Put the raid sets under the control of the sanmelody servers (this is called “protect”ing) 15 11.) Set up network managed volumes, create virtual volumes AFTER and then sort out replication between the two arrays (as detailed below). Serve up the primary luns to both esx servers and make sure that they can see it. 12.) Serve up the failover luns to both esx servers and make sure they can see it (remember the lvmresignature option see below as this is very important). 13.) Make sure failover paths are correct and set up multipathing properly making sure the iscsi channels are linked to correct ip addresses (i.e. network adapters) and that they iscsi channels are mapped correctly. **Setting up the Microsoft iscsi initiator is a prerequisite for high availability. You need to set it up so that both disk arrays are pointing to each other ** In order to set up replication between the sans you must do the following….The steps to do these in san melody are in the sanmelody guide but this guide should be better. Always remember to apply your configurations after doing even the smallest change. 1.) Set up your raid sets on both arrays exactly the same. You really do not want to get involved in the mess of mirroring data between different sized volumes and raid sets. SO set your raid sets up but do not partition them yet. 2.) You should initialize all new volumes on both san1 and 2 using the windows disk manager after creating the raid sets. Do not create new partitions on both sans for each volume as you should be assigning the raw disks to the network managed volume pools. Now go into the mmc for datacore and click the storage server in the left hand side window. Now right click the application server in the right hand side and protect all volumes (do this on both servers) remember to apply the configuration and go out of the mmc on each san. 3.) You can now partner both sans. First do the following: Go to Computer Management/Services and Applications/Services. Select DataCore Storage Domain Services service. Right-click DataCore Storage Domain Services and select Properties. On the Log On tab, click This Account. Click the Browse button to browse for an existing account. Click the Advanced button. Click the Find Now button. Select the account you wish to use for the DcsSDS account. Click OK, OK, and OK. Enter the password you used when you created the account. Note The logon and password associated with DcsSds on this computer must match the logon and password associated with DcsSds on the partner computer. Click OK. You will see a confirmation message. Click OK. Right-click DataCore Storage Domain Services and select Stop. 16 Right-click DataCore Storage Domain Services and select Restart. 4.) On one storage server (for example, StorageServer1), go to the DataCore SANmelody snapin tree, and select Storage Server. Right-click on the server name in the top right panel and select Partner. This opens the Partner tab of the Storage Server Properties window. Select Allow Incoming Partner and click OK. You will see a message telling you to close Computer Management. Do not open it againuntil you add the partner on the other storage server Now go to the other storage server (StorageServer2) and repeat steps 1 and 2. The Partner tab of the Storage Server Properties window displays. Select Add Partner and enter the hostname of the partner SANmelody storage server(StorageServer1). Click OK. You will see a warning "Note that this operation will automatically apply your current configuration." You must click OK to proceed. Your partner name appears in the partner information area. You will see a progress bar when you are establishing a partnership. Notes   If the partner is valid and accessible (licensed for the High Availability option and available), you will get a message that the partnership is authorized. Click OK. If the partnership is denied, you will get a message that the server is Unreachable, No License, Refused (already has a partner) Click OK to complete the partner definition process. Click the configuration Apply icon on the toolbar. You will see a message that the configuration was successfully applied. Click OK to finish the apply step. 4.) Go into the sanmelody mmc and create a nmv pool (you create a pool then assign a raw disk and then create a network managed volume) under the nmv option under disk manager in the right hand window i.e. right click and create a new nmv pool (I have created 3, one for each raid array I created) and assign the raw disks (each raid set should correspond to one raw disk) to the network pool by right clicking(i.e. one raw disk to one network pool and make sure you get the correct raid sets!). Remember to right click the pool after you have added the raw disk and add network managed volume (this adds the nmv to the sanmelody storage server, do this on both sans BEFORE creating your mirrors and virtual volumes!!!). Be wary of adding new raid sets later on to nmv 17 pools because you can essentially spread your nmvs over different raid sets and you may inadvertently get half a virtualmachine on one raid set and another virtual machine on another raid set). My idea would be to set up new nmv pools to stop this happening though it SHOULD work fine . You should set up nmvs on both sans (give them suitable names but not all the same on each san or matching pairs) otherwise you will get seriously confused!). Always make sure that you right click the pool and “add network managed volume” for each pool set once you have assigned a raw disk to the nmv otherwise you will not be able to create a virtual volume based on the network managed volume (you will also not be able to set up a mirror correctly if you only add one sans nmvs!). Make sure you apply the configuration in the mmc and exit the mmc then you can go back in. 5.) If you go back into the mmc and click storage server you should now see three network managed volumes. If they are showing as red make sure that you click apply config. Then exit the mmc and go back in. once you have linked your disks to your nmvs…. 6.) You can now create a virtual volume on san1 based on san1’s nmv’s (pool1 and pool2 etc). DO this by right clicking the nmv under the storage server in the mmc and create a virtual volume. Do all three nmv’s on san1 (or however many nmv’s you created) Do not do this for san2!. 7.) Now you can right click the virtual volume on san1 and select mirror and choose the nmv that it will be mirrored to on san2. Make sure that it is a 3 part AP mirror.. Click apply. Do note however that in my setup the nmv pool name changed on san2 to “pool1” after being partnered. NMV replication is a pig by the way. You may need to set it up by trial and error with a couple of reboots of the san sometimes. 8.) Now you need to set up the iscsi links between the esx servers and the sanmelody servers (see step 10, step 9 is an overview of what you are about to do) 9.) Now you can add your application servers to the datacore snapin. So click on the application server icon and right click and add new application server and select vmware from the drop down and give it a name. Create another one on the same san and then do the same on san2 (i.e. for me I have 2 application server showing on both sans). 10.) Now you need to link the virtual volumes to both esx servers on both sans rd You can now serve up the mirrored virtual volumes to the esx servers. I have three. Each mirrored virtual volume equates to one nmv which equates to one nmv pool which equates to one raw disk which equates to one raid set (phew now that is abstraction for you!!). Go into the datacore mmc and go into the virtual volumes tab. Right click a virtual volume and link it to an application server. 18 The above mapping shows that you have successfully linked the virtual volume to both esx1 and esx2 (that is the application server you have just created on san1 NOT the real esx server!!! This is the next step!). Not only that but you have also linked the mirror of this virtual volume to both esx1 and esx2. Do note that all the luns for this particular virtual volume are 0!! (we are only dealing with 1 lun here, when you create another lun, say lun 1 on another raid10 set that is mirrored between san1 and san2 and is displayed to both esx1 and esx2 it needs to be consistently lun 1) This is important for every multipath mirrored virtual volume. If you add another multipath mirrored volume you should make sure the lun is set to a different number and it will normally do this automatically (you can check by right clicking the virtual volume or the application server and looking at its properties). BUT the lun for that nmv must be the same on each VMWARE server. You will need to add in the alternate paths and scan for storage changes on the esx servers when you add new mirrored network managed virtual volumes. You do this by going to virtual center and then configuration tab and storage then click rescan. If the new lun is just that, i.e. new!, then you can format it with the vmfs file system. Absolutely make sure this is the correct LUN! Sometimes existing luns with vmfs partitions on do not show (try rebooting the vmware servers). Now link the application server channel to the esx server: Go to the virtual infrastructure client and go to the iscsi initiator by clicking storage adapters for one of the esx servers (you need to do this for both by the way), scroll down and highlight iscsi then click properties and configure and enable it. Then go to the dynamic discovery tab and add in the ip addresses of the two disk arrays (after you have set up the san melody volumes). Do not change the port (remember to take off the firewall settings on the san melody server). Now click the rescan link in the top right hand corner. By doing this it will 19 enable an option on the san melody server. You now need to go back to the san melody server and go to the application server node and right click it and go to properties. Go to the channel tab and now you should see the iscsi hba of the esx server in one of the tabs, you need to move this to the window above and click apply). Click apply in the mmc tab. Now go into both esx servers using the virtual center client. Go to the configuration tab and click storage adapters and do a rescan (make sure you have added in the ip of both sans into the properties section of the storage section in esx.first! Once the luns show up go to the storage (scsi san and nfs) link. In here you can add storage. Make sure you give it a name specific to where the storage is to make it easier for administrators. Now… If you are just rescanning an existing vmfs partition you should not be given the option of formatting the drive with the vmfs file system (in fact it should just show up after a rescan but sometimes doesn’t, see later). If this is a new drive then it will ask you to do this. Once you have formatted the partition for esx1 (for example) if you rescan on esx2 the vmfs partition you just created should automatically show up (if it doesn’t then you will not be able to vmotion or failover in the event of an esx failure). If you are using network managed volumes on the san will show up as 2tb in esx storage (please read the glossary section at the beginning of this document explaining why this is so). Just a note on partitioning and formatting…. You should read the comments on the vmware forums about what sizes to set up etc as it can affect the amount of space you can use on a lun. Check your alternate paths: Go to vmware infrastructure client and click storage on the desired esx server. Now right click the storage that you want and click properties and then manage paths. You should see an 20 active path(to one of the sans) and a failover path (to the other san). If you don’t see this then check you iscsi channel mappings. Once these are correct you should be set up for a san link failure between the esx servers, high availability (if one of the esx servers die the other esx server will start the vms) and vmotion. You can test this manually by going into the vi inf client and then config and then storage and then click properties. Make sure both paths are enabled and then disable one of them by clicking change. OH and another thing you should be using MRU (most recently used for your paths) and you should add the following line into the advanced settings (it is under config then advance settings and then disk in the vi inf client (on both vmware servers) : Disk.sandeviceswithapfailover SANmelody : (do the last two things before trying the failover!) 21 Setting up a high availability cluster/vmotion setup in vmware At this stage in the set up lets just recap where we are… 1.) You have 3 raid sets replicating (as nmv’s) between both disk arrays (you may only just have one called lun0 however if you are feeling adventurous you should create another and mirror it then serve it up to the esx servers. 2.) Both esx servers can see all raid sets (i.e. esx1 can see all raid sets on san1 and all raid sets on san2). Sanmelody actually works in a weird way. When you “partner” 2 sans there is no primary and secondary san. You can therefore have for example lun1 live on san1 and lun2 live on san2 (if lun1 is live on san1 then lun1 on san2 is secondary etc). 3.) you have alternate paths set up. What we now need to do is test the vmotion setup and the cluster failover set up and then finally sort out the failover ip configuration. You must set up a cluster before you can use vmotion! Create a cluster: At the moment you have only set up two individual servers under the virtual center (i.e. esx1 and esx2) and put them into a datastore. You now need to set them up in to a cluster so that you can use vmotion and the high availability option of vmware. In order to do this right click the datastore and add a new cluster. Tick vmware ha and drs. (make sure you use manual drs), accept defaults in next page click next and then finish. You know need to add the hosts to the cluster. You do this by right clicking the cluster and adding the hosts as necessary. At this point you should now be able to test your vmotion and the high availability. Obviously add your servers into the cluster How to Vmotion between esx servers: Considering how difficult this is to set up, actually vmotioning is very simple. All you need to do is right click the virtual machine that you wish to vmotion and then go to migrate. You just choose the destination esx server and set the priority to high and click finish. I would recommend running a ping here to the virtual machine. You will see a blip and then it should be moved. Having said that there is also a status bar on the bottom so you can see how things are going. 22 Vmware high availability: this basically works by restarting the virtual machine on the other esx host if there is an esx failure. To test this just pull the network cables out of the back of esx1!. At this point you should now be able to vmotion your servers on the 10.0.0.0 network whilst logging in to your virtual machine on the 128.1.0.0 network (if you have created any vms!). Any probs with HA not working (errors in the infrastructure client. Just right click the esx host in the left hand side and click reconfigure for HA, you may need to reboot the esx server or stop/start the virtual centre service on the virtual centre pc, see known problems for more about this) If you have any problems bear in mind layer two loops on the switches and that you may need to reboot the esx servers and the virtual center. You will probably need to do a rescan on the storage adapters as well. Once you have changed the ips over you should test the high availability and vmotion. Your virtual switches should look like this on esx 1 (yes you will need to create a new virtual switch on both esx1 and 2. You will need to have the vmkernel on the 10.0.0.0 virtual switch and the vmnetwork on the 128.1.0.0 virtual switch. This is because you want vmotion and high availability to run on the 10.0.0.0 (vmotion and high availability use the vmkernel). You will have to add a vmkernel to the second virtual switch and take it off the original virtual switch (basically you want to move the vmkernel from the 128.1.0.0 vswitch to the 10.0.0.0 vswitch). (keep the vlans as default). You will also need to make sure that you have added a vm 23 network to each 128.1.0.0 virtual switch so that you can put the actual virtual machines on to the live 128.1.0.0 network. If you have any problems here you may need to do a bit of juggling in terms of deleting components of the original virtual switch and then adding them in to the second vswitch that you created (linked to the 10.0.0.0 network). You will also need a service console on each otherwise problems may arise. To add a switch click add networking. To add components to the vswitch click properties and click add or edit to add components to the vswitch like service console, vmkernel, network adapter etc. I believe you can team nics in this area by simply adding a second adapter. As yet I haven’t tested this as I don’t have any spare nics. esx2: 24 Performance testing Generally speaking it is the disks in a machine that cause the bottleneck. I have had to devise ways of monitoring the performance of the sans to see how they will handle and what configuration is best. This has led me down a path of learning about iops (input/output per second), network latency, memory access and raid arrays. The way that I have monitored the performance was quite easy. In fact it helped immensely that there was a competition on the vmware community regarding how fast your sans and esx servers were. The way to do it was using iometer (search google and install it). There was a default config someone set up in a performance thread on the vmware community forum. Obviously in my config we are not going to see world class performance but we can see some interesting stats. I have set up the arrays with 2 raid 10 sets. The first raid 10 set (lun0) has 4 disks(sata 7,200) and the second raid 10 set has 6 disks. Bare in mind that most servers read rather than write. I decided to split them up in to two for the fact that one could be for backup and the other for the vms. More spindles = better read performance. Not sure yet if I should go down the one mammoth raid 10 set or two raid 10 sets yet. Ideally I want three fast raid 10 sets but I would be limited to 4 disks in each raid 10 set which means low iops performance. The final raid array config will be based on performance from a normal work day perspective and from a backup perspective. I am seeing a roughly 20 percent increase in performance when using 6 spindles rather than 4. I therefore would like to run the whole sanmelody server on one raid 10 set, however I can’t make that choice at this point because I do not know how I am going to build the set up based upon snapshots, backups and live vms. It is usually best practise to split your vms over different raid sets depending on how intensive they are. The other part of the performance testing is to monitor current live servers. You can use perfmon to do this and track the disk reads and disk writes which is measured in iops. You can then compare it to your sanmelody servers (create a vm in esx and install iometer in it and do some testing for the raid set the vm is on). This will tell you what sort of amount of servers you can put on a lun in terms of disk access. 25 Backing up virtual machines When it comes to backing up virtual machines there are a number of methods at our disposal. The first one is to use the vmware consolidated backup software. This essentially is a proxy backup machine that takes the strain off the live virtual machine by mounting the virtual machine disk onto another server and backing it up from there. However it does require another machine to use as a proxy (a 2003 server only!!). Another alternative is to use esxpress free edition which will do full backups every night of every vm (alternatively we could set it up weekly during a time when no-one is on the machines) and then do backups from within the virtual machines as usual (for file level backup). The other alternative is to use a vmperl sript which will ftp the vmdk files to a nas ftp server.In the end I opted to use esxpress free edition. I have therefore bought a buffalo terastation 1tb machine which has an inbuilt ftp server. The idea being backups will be done from the raid 10 sets to the raid 0 set. Then copied from the raid 0 set to the terastation. This can all be done by using esxpress and a bit of scripting (see later on). The alternative is to use the vmbk script (another option you could go down, I tested this route so for future reference if you need to use in and install it you can do the following) Installing esxpress: You will need to use winscp or putty to do this. Download esxpress free from the esxpress website. Login to the esx servers using puttyssh and go to the root directory. Create a directory called esxpress. Then copy the esxpress files from your pc to the esx servers using winscp to the directory you just created. Follow the install guide. Now set up the backup software using the vba (you need to start the gui from the command line menu on esx. Therefore putty into esx1 then login as root then type phd (press enter) then choose the start gui option from the menu, wait and then go into the virtual centre client. The esxpress vba should now start up and you can set it up. Fairly easy to do. Just specify what you want to backup and where it should go (either to a lun or the terastation, remember you will need the username and password for ftp). You will then need to test the backups and restore 26 Restoring can only be down by puttying into the esx server and typing “phd” then go to the restoration menu and follow the prompts: Restoring can only be down by puttying into the esx server and typing “phd” then go to the restoration menu and follow the prompts. Setting up backups: Use the ssh utility “putty” to ssh into the esx server. Then type “phd” and choose the start gui option from the menu. This will start the esxpress virtual appliance and within a few minutes you will be bale to see it in action in the virtual infrastructure client:- Click esxpress v3 configuration: 27 Within here you can set up your daily full backups for your vms and the ftp server (i.e. the buffalo terastation). To backup powered off vms search for the option under “options/advanced options” To restore a server you need to do the following: Restores can only be done from the “phd” command line menu. So to test, for example…. Go into ssh “putty” and putty into esx1. type phd and choose the restore option from the menu 28 Then choose restore through the esx console option, now choose restore by vm name and then choose full restore (not a delta!). restore it to the lun that you want and sit back…. Once restored you need to go back into the virtual inf client and right click the cluster and create a new virtual machine, choose the custom menu option go through the wizard until you get to the select a disk option. This is where you specify that you want to use an existing disk. Click next and then choose the browse datastore option and then choose the vmdk file that you just restored 29 i.e. Click next and next again (don’t choose independent disks) and click finish. Power it on and see if it works! If you get the lsi or bus logic option click no and it should all boot up nicely! 30 Other Tasks: Maintenance, logs, snapshots patching, adding storage, setting up email alerting, remote command line access Connecting to the esx server command line remotely: Use a SSH client like putty. (download it) then point it to the ip of the esx server but first you need to allow root access through ssh : So log in at the console of the esx server (normal root username and password) edit the file /etc/ssh/sshd_config and change "PermitRootLogin" to yes (after that "service sshd restart") you should now be able to login using ssh from any xp machine with the ssh putty software installed. If you have any problems with ssh or ftp uploading etc you should check the vmware forums for how to turn off the firewall. Copying files to the esx servers from a windows machine You must have set up access using ssh (using putty or something similar) so that you can login. Also you may need to amend the firewall settings on the esx servers. you can do this by going in to the vi client selecting your esx server and clicking on the configuration tab and clicking on the security profile link. By default the esx servers do not allow root level remote access connections so you have to edit a configuration file. Follow the steps above for accessing an esx command line remotely. Once this works you must download winscp (from the interweb). This will allow you to copy files using ftp from your local windows machine to your esx server. It is fairly self explanatory with your c drive on the left and your esx server on the right:- 31 How to upgrade from version 3.0 to 3.01 and install any other patches. The CD you get from vmware is version 3.0 and vc 2.0 If you reinstall you will need to install the latest versions from the cd’s I downloaded from the website or install off of the old originals and then upgrade. The upgrade is relatively easy. Just pop in the upgrade cd for version 3.01 for the esx servers and follow the wizard accepting the default values. The same applies for virtual centre 2.01 however you need to backup your sql database first, run the upgrade and then restore it. If you don’t do this you lose your vc settings. However, this is not the end of the world as the esx servers keep there settings so all you have to do is create a new data store again, create a cluster with drs and ha (setting drs to manual) and add the esx servers back in to the cluster. You may need to reboot the vc server as it will do its usual of not handing out the licenses to the esx servers so you wont be able to turn on any vms. Patching esx servers: Patching esx is not easy at the moment. Apparently they are releasing a new patching tool in the next upgrade. But we will have to wait and see. For the moment go here to see what the latest patches are: http://www.vmware.com/download/vi/vi3_patches.html Snapshotting at the SAN Level Sanmelody has a snapshot feature though it is not particularly useful at the moment. Basically it will snapshot a lun (or raid set/virtual volume) to another lun. Which means you have to have a virtual volume of equal size free with nothing on it to be able to do snapshots. Due to the complexity of setting this all up and the problems inherent with it I have given it a miss. Basically, you can “snapshot” vmfs filesystems residing on one lun to another empty lun of the same size. But the transfer rate is 30 meg a second where as my backup procedure is 25 meg a second. So not a lot in it. The main problem with doing it is that you need more disks in the sans which we don’t have. We “could” set it up in theory but it would involve vreating more than one vmfs partition on a raid set which would producer contention on the hard disks for vms during heavy usage. This might be worth reconsidering at a later date. 32 Log files: Log files are not necessarily easy to read in vmware however they are useful when it comes to giving you some sort of errors. The best thing to do is post them up on the vmware community forum or give them to the vmware support team. For now all you really need to know is that the logs are situated in \var\log on the esx hosts Just log in using putty ssh and vi the following files to get an idea of any errors Vmkwarning Vmkernel Vmksummary There are some other log files in there but I am not sure what they refer to! Setting up SNMP: This is a fairly straight forward process however at the moment solar winds (snmp monitoring software I use) cannot take advantage of all the options because the mib situated in virtual centre cannot be uploaded into solar winds to define all its counters. This will be fixed when solar winds is upgraded to version 8. For the moment you need to enable the snmp port on the esx hosts so go into virtual inf client and click configuration and then security and tick the snmp allowed box, 33 click ok. Now login to each esx host and go to the \etc\snmp directory and edit snmpd.conf. You need to change the trapsink line to look at solarwinds.lloydmaunder.co.uk and then save and quit. Now you should start the snmp demon by typing service snmpd start and then to make it always start on reboot type “chkconfig snmpd on”. Now all you have to do is add the esx servers to solar winds system manager. Email alerting: To set up email alerting for the raid sets go into the 3ware manager and go to the email configuration tab. Set up is fairly self explanatory To set up esxpress email alerts to alert you on successful completion of backups (or failures) start the esxpress gui from the commandline on an esx command line by typing phd and then choose the start gui option. Now go into the vi inf client and select the virtual appliance for esxpress and go to the email configuration tab and set it up as appropriate. Backing up and restoring the virtual centre msde: Go into the virtual centre and go into a command prompt and type the following to back up To restore a database is fairly simple. Bare in mind you can back up and restore using enterprise manager as well. Search the web for how to restore an msde backup. Should be something like the above. Also it is not the end of the world as we don’t have a complex virtual centre settings. If you cant restore just recreate the cluster as per the instructions in this guide and add the esx servers back in plus set up the rules for ha and drs. 34 Links: Backups: Virtual Machine Backup Guide - http://www.vmware.com/pdf/vi3_vm_backup.pdf Consolidated Backup in Vmware Infrastructure 3 http://www.vmware.com/pdf/vi3_consolidated_backup.pdf Vmware Consolidated Backup Best Practices, Tips and Tricks http://download3.vmware.com/vmworld/2006/bct4540.pdf Backup and Recovery of Virtual Servers http://download3.vmware.com/vmworld/2006/mdc9870.pdf Hot Backups and Restores - http://download3.vmware.com/vmworld/2006/tac9816.pdf Non-disruptive Backups of Vmware Environments Using Veritas Netbackup http://download3.vmware.com/vmworld/2006/tac9912.pdf Vmware Consolidated Backup for Disaster Recovery http://download3.vmware.com/vmworld/2006/labs2006/vmworld.06.lab01-VCBPRESENTATION.pdf Perl Backup Script for Vmware ESX - vmbk.pl ( free ) - http://www.vmts.net/vmbk.htm ESX Ranger ( not free ) - http://vizioncore.com esXpress ( not free ) - http://esxpress.com Backup scripts - http://www.rtfm-ed.co.uk/?p=23 and http://www.rtfm-ed.co.uk/?p=156 Shell script - http://www.vmware.com/community/thread.jspa?messageID=475244񴁬 Perl script - http://www.vmware.com/community/thread.jspa?threadID=70253&start=0&tstart=0 Oracle backups http://www.vmware.com/community/thread.jspa?threadID=76746&messageID=601703#601703 Vmware Consolidated Backup Technology: Today and Future http://download3.vmware.com/vmworld/2006/bct4539.pdf Implementing Effective Backup Strategies for Disaster Recover http://download3.vmware.com/vmworld/2006/bct9502.pdf Books/Documentation/Learning Resources: http://www.lulu.com/content/712361 http://www.vi3book.com http://www.vmguru.com/ Vmware VI3 documentation - http://www.vmware.com/support/pubs/vi_pubs.html Configuration Maximums for VMware Infrastructure 3 http://www.vmware.com/pdf/vi3_301_201_config_max.pdf Vmworld presentations - http://www.vmware.com/vmtn/vmworld/ ESX on non-supported hardware to learn with - 35 http://www.vmware.com/community/thread.jspa?threadID=77560 Citrix Optimizing Citrix Presentation Server on Vmware ESX Server http://download3.vmware.com/vmworld/2006/med0115.pdf Citrix and Vmware - http://download3.vmware.com/vmworld/2006/tac9728.pdf Cloning: Newsid - http://www.microsoft.com/technet/sysinternals/Security/NewSid.mspx Sysprep - http://www.microsoft.com/technet/prodtechnol/winxppro/deploy/duplication.mspx Sysprep - http://www.vernalex.com/guides/sysprep/ Clustering: Setup for Microsoft Cluster Service - http://www.vmware.com/pdf/vi3_vm_and_mscs.pdf Converter/P2V: Boot.ini file - http://mirror.href.com/thestarman/asm/mbr/bootini.htm Guide to P2V 2.x - http://www.rtfm-ed.eu/docs/vmwdocs/admin-p2v-2.0.pdf Vmware P2V Assistant Best Practices - http://download3.vmware.com/vmworld/2006/tac9886.pdf Converter tutorial http://www.petri.co.il/virtual_convert_physical_machines_to_virtual_machines_with_vmware_con verter.htm Introducing the Next Generation of P2V: Vmware Converter 3.0 http://download3.vmware.com/vmworld/2006/tac9453.pdf Converter FAQ - http://www.vmware.com/products/converter/faqs.html Converter Manual - http://www.vmware.com/pdf/VMware_Converter_manual.pdf Converter Release Notes http://www.vmware.com/support/converter/doc/releasenotes_conv3.html Converter Data Sheet - http://www.vmware.com/pdf/converter_datasheet.pdf Converter download - http://www.vmware.com/download/converter/ Import of physical machine fails http://www.vmware.com/community/thread.jspa?messageID=567860򊨴 Ultimate P2V - http://www.rtfm-ed.co.uk/?page_id=174 EZ P2V - http://www.ezp2v.net/ VMware P2V and Virtual Machine Importer http://download3.vmware.com/vmworld/2006/labs2006/vmworld.06.lab09-P2V.VMImporterMANUAL.pdf What machines should not be converted - 36 http://www.vmware.com/community/thread.jspa?threadID=73357 Remove non-present devices from converted system http://www.oreilly.com/catalog/winxphks2/chapter/hack116.pdf http://support.microsoft.com/kb/315539 http://www.vmware.com/support/p2v21/doc/updatedevices7.html Linux conversion - http://virtualaleph.blogspot.com/2007/04/cloning-linux-machine-withconverter.html Linux conversion http://www.vmware.com/community/thread.jspa?messageID=562582򉖖 Domain Controllers/Active Directory: Virutalization of Active Directory http://www.vmware.com/community/thread.jspa?messageID=352424񖂨 Considerations when hosting Active Directory domain controller in virtual hosting environments http://support.microsoft.com/kb/888794 Virtualizing a Windows Active Directory Domain Infrastructure http://download3.vmware.com/vmworld/2006/tac9710.pdf How to detect and recover from a USN rollback in Windows 2003 Server: http://support.microsoft.com/kb/885875/ How to detect and recover from a USN rollback in Windows 2000 Server: http://support.microsoft.com/kb/875495/ P2Ving Domain Controllers http://www.vmware.com/community/thread.jspa?messageID=390630 ESX: Disabling Web access on the ESX Server http://www.vmware.com/community/thread.jspa?threadID=59508&tstart=0 Using SCP to Copy Files to or from an ESX Server System - http://kb.vmware.com/kb/1918 Using SCP to Copy Files to or from an ESX Server System http://www.vmware.com/community/thread.jspa?messageID=599300򒔄 ESX Server 3.0 Tips and Tricks - http://download3.vmware.com/vmworld/2006/tac9567.pdf Change VM swap file path - http://www.vmware.com/community/thread.jspa?messageID=530426 Change VM swap file path - http://www.vmware.com/pdf/vi3_esx_resource_mgmt.pdf (Page 141) Vmimages obsolete - http://www.vmware.com/community/thread.jspa?messageID=503192 Vmimages obsolete - http://www.vmware.com/community/message.jspa?messageID=530228 3.0.1 restart bug http://www.vmware.com/community/thread.jspa?messageID=597633򑺁 37 3.0.1 restart bug - http://www.vmware.com/community/thread.jspa?threadID=75462&tstart=0 General: Server product comparison - http://www.vmware.com/products/server_comp.html Introduction to ESX 3.0.1 and VirtualCenter 2.0.1 - http://www.vmware.com/pdf/vi3_intro_vi.pdf Tips and tricks for implementing infrastructure services on ESX server http://www.vmware.com/pdf/tips_tricks_infrastructure_services.pdf Virtual Infrastructure Implementation Best Practices From A to Z http://download3.vmware.com/vmworld/2006/adc0051.pdf Vmware Infrastructure 3 Customer FAQ - http://www.vmware.com/pdf/vi3_customer_faq.pdf Server Consolidation with VMware ESX Server http://www.redbooks.ibm.com/redpapers/pdfs/redp3939.pdf Virtual Machine Cost Calculator spreadsheet http://www.ronoglesby.com/downloads/vmoglator1.1.xls TCO Calculator - http://www.vmware.com/products/vi/calculator.html? HA/DRS: Automating High Availability (HA) Services with Vmware HA http://www.vmware.com/pdf/vmware_ha_wp.pdf Effective DRS and HA in Production - http://download3.vmware.com/vmworld/2006/tac9413.pdf Resource Management with Vmware DRS - http://www.vmware.com/pdf/vmware_drs_wp.pdf Resource Management in Vmware ESX Server 3 http://download3.vmware.com/vmworld/2006/tac9726.pdf Intel & AMD with HA & DRS http://www.vmware.com/community/thread.jspa?messageID=526615 Hardware: Vmotion CPU Compatibility for Intel processors - http://kb.vmware.com/kb/1991 Vmotion CPU Compatibility for AMD processors - http://kb.vmware.com/kb/1992 Vmotion - How to override CPU masks due to cpu mismatch - http://kb.vmware.com/kb/1993 Vmotion requirements - http://www.vmware.com/pdf/vi3_admin_guide.pdf (Chapter 14) Vmotion between Apples and Oranges: Understanding CPU Compatibility Restraints for Vmotion - http://download3.vmware.com/vmworld/2006/tac1356.pdf Vmotion and CPU Compatibility http://www.vmware.com/community/thread.jspa?messageID=452231񮚇 Vmotion Info tool - http://www.run-virtual.com/?page_id=155 HP Management Agents for VMware ESX Server 3.x version 7.7.0 http://h18023.www1.hp.com/support/files/server/us/download/26407.html 38 Vmotion and VM Compatibility Matrix for Dell PowerEdge Servers http://www.dell.com/downloads/global/solutions/vmotion_compatiblity_matix.pdf Vmotion and VM Compatibility Matrix for HP Proliant Servers ftp://ftp.compaq.com/pub/products/servers/vmware/vmmotion-compatibility-matrix.pdf HP Vmotion Compatibility Matrix http://h18004.www1.hp.com/products/servers/software/vmware/hpvmotion-compatibilitymatrix.html Production Consolidation using VMware and the AMD Opteron Processor http://download3.vmware.com/vmworld/2006/adc9743.pdf IBM Insights in Sizing Hardware for Virtualization http://download3.vmware.com/vmworld/2006/tac4057.pdf HP Virtualization Management Software http://h18013.www1.hp.com/products/servers/proliantessentials/valuepack/vms/index.html 10K vs. 15K & iSCSI - http://www.vmware.com/community/thread.jspa?threadID=75202 To vSMP or not to vSMP http://www.vmware.com/community/thread.jspa?messageID=298150񈲦 Best Practices using Vmware Virtual vSMP http://www.vmware.com/pdf/vsmp_best_practices.pdf Hardware recommendations to build a cheap ESX server http://www.vmweekly.com/articles/hardware_recommendations_to_build_cheap_esx_server/1/ White box/Home ESX system http://www.vmware.com/community/thread.jspa?messageID=620124򗙜 Licensing: Vmware licensing page - http://www.vmware.com/download/licensing.html Server-based License File Checker - http://www.vmware.com/checklicense/ Pricing, Packaging & Licensing Overview - http://www.vmware.com/pdf/vi_pricing.pdf Install License File - http://www.vmware.com/community/thread.jspa?threadID=76518 Microsoft Virtual Machine Technology FAQ http://www.microsoft.com/licensing/highlights/virtualization/faq.mspx Windows Server Virtualization Calculator http://www.microsoft.com/windowsserver2003/howtobuy/licensing/calculator.mspx Licensing Microsoft Server Products with Virtual Machine Technologies http://download.microsoft.com/download/6/8/9/68964284-864d-4a6d-aed9f2c1f8f23e14/virtualization_whitepaper.doc Linux: Getting a DHCP Address in a Red Hat Linux Virtual Machine - http://kb.vmware.com/kb/977 39 Windows to Linux roadmap - http://www-128.ibm.com/developerworks/linux/library/lroadmap.html Windows to Linux roadmap post on Vmtn http://www.vmware.com/community/thread.jspa?threadID=77279 Install Vmware Tools in a Linux VM under ESX Server 3.x - http://kb.vmware.com/kb/5242329 From DOS/Windows to Linux HowTo - http://www.tldp.org/HOWTO/DOS-Win-to-LinuxHOWTO.html Universal Command Guide for Operating Systems - http://www.allcommands.com/ Memory: Host Mem and Guest Mem % http://www.vmware.com/community/thread.jspa?messageID=620581򗠥 http://www.stanford.edu/~stinson/cs240/cs240_1/revs/esx.txt The Role of Memory in ESX Server 3 - http://www.vmware.com/pdf/esx3_memory.pdf Memory resource management in Vmware ESX Server http://www.vmware.com/pdf/usenix_resource_mgmt.pdf Understanding memory usage in Windows 2000 http://www.microsoft.com/technet/prodtechnol/windows2000serv/tips/w2mem.mspx Operating System Extensions to Support Host Based Virtual Machines http://www.eecs.umich.edu/techreports/cse/2002/CSE-TR-465-02.pdf Memory performance - http://www.vmware.com/community/thread.jspa?threadID=65293 Memory Shares/Limits/Reservations http://www.vmware.com/community/thread.jspa?threadID=73411 RAM, Virtual Memory, Pagefile and all that stuff - http://support.microsoft.com/?kbid=555223 Understanding the Impact of RAM on Overall System Performance http://www.windowsnetworking.com/articles_tutorials/Understanding-Impact-RAM-OverallSystem-Performance.html RAM, Virtual Memory, Pagefile and all that stuff http://members.shaw.ca/bsanders/WindowsGeneralWeb/RAMVirtualMemoryPageFileEtc.htm Pagefile thread http://www.vmware.com/community/thread.jspa?threadID=79080&messageID=617876#617876 Resizing Memory With Balloons and Hotplug - https://ols2006.108.redhat.com/reprints/schoppreprint.pdf Networking: VMware ESX Server 3 802.1Q VLAN Solutions - http://www.vmware.com/pdf/esx3_vlan_wp.pdf Networking Virtual Machines - http://download3.vmware.com/vmworld/2006/TAC9689-A.pdf Networking Scenarios & Troubleshooting - http://download3.vmware.com/vmworld/2006/tac9689- 40 b.pdf Duplex Mismatch Primer - http://www.happyrouter.com/content/view/32/1/ 3rd party software and the new Service Console Firewall http://kb.vmware.com/KanisaPlatform/Publishing/302/2143_f.SAL_Public.html Firewall and Netbackup http://www.vmware.com/community/thread.jspa?messageID=477399񴣗 DMZ & VLANs - http://www.vmware.com/community/thread.jspa?messageID=347532񔶌 ESX & DMZ - http://www.vmware.com/community/thread.jspa?messageID=233918𹆾 ESX & DMZ - http://www.vmware.com/community/thread.jspa?messageID=344471񔆗 ESX VM’s in the DMZ http://www.vmware.com/community/thread.jspa?forumID=21&threadID=19402&messageID=2223 99#222399 Second Service Console NIC http://www.vmware.com/community/thread.jspa?messageID=536518򂿆 NIC teaming and VLAN trunking - http://blog.scottlowe.org/2006/12/04/esx-server-nic-teamingand-vlan-trunking/ Which NIC is used in cold migration http://www.vmware.com/community/thread.jspa?messageID=555404򇦌 Esxcfg-firewall - http://72.14.205.104/search?q=cache:do0cmwxcWsJ:download3.vmware.com/vmworld/2006/labs2006/vmworld.06.lab05-SECURITYMANUAL-APPENDIX.pdf+esxcfg-firewall&hl=en&ct=clnk&cd=6&gl=us Service Console and Vmotion NIC sharing http://www.vmware.com/community/thread.jspa?messageID=146348𣮬 Configure multiple IP addresses to a NIC in Linux - http://handsonhowto.com/virt.html Configure a static MAC address for a VM - http://virtrix.blogspot.com/2007/04/vmwareconfiguring-static-mac-address.html Rename a vSwitch - http://www.vmware.com/community/thread.jspa?threadID=80075&tstart=0 Rename a vSwitch http://www.vmweekly.com/ask_an_expert/how_can_I_rename_a_virtual_switch/1/ Changing the MAC address of a virtual machine http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId =507 Patches: Patch Management for ESX Server 3 - http://www.vmware.com/pdf/esx3_esxupdate.pdf Download - http://www.vmware.com/download/vi/index.html ESX patches - http://www.vmware.com/download/vi/vi3_patches.html#c4310 Virtual Center 2.0.1 Patch2 - http://www.vmware.com/support/vi3/doc/vc-201-200702-patch.html 41 Autopatching ESX host - http://virtrix.blogspot.com/2007/03/vmware-autopatching-your-esxhost.html Patching and updating Vmware ESX Server 3 http://download3.vmware.com/vmworld/2006/tac9858.pdf Automated download of ESX patches http://www.vmware.com/community/thread.jspa?messageID=600341 Patch Script: - http://www.vmprofessional.com/material/esx-autopatch.pl Pre-compiled patchlist.txt - http://www.vmprofessional.com/material/3.0.1/patchlist.txt and http://www.vmprofessional.com/material/3.0.0/patchlist.txt Patch order - http://www.vmware.com/community/thread.jspa?threadID=80531 amd http://www.vmware.com/community/thread.jspa?threadID=80296 Performance: VMware ESX Server guest performance tips Pt .1 http://searchservervirtualization.techtarget.com/tip/0,289483,sid94_gci1231666,00.html VMware ESX Server guest performance tips Pt. 2 http://searchservervirtualization.techtarget.com/tip/0,289483,sid94_gci1231917,00.html VMware ESX Server guest performance tips Pt. 3 http://searchservervirtualization.techtarget.com/tip/0,289483,sid94_gci1235138,00.html Top 12 ways to Improve Guest Server Performance under VMware ESX Server http://download3.vmware.com/vmworld/2006/mdc9700.pdf Vmware ESX Server 2 Performance Tips and Tricks http://www.vmware.com/pdf/esx_performance_tips_tricks.pdf Performance Tuning for VI3 - http://download3.vmware.com/vmworld/2006/tac9872.pdf Performance Troubleshooting http://download3.vmware.com/vmworld/2006/labs2006/vmworld.06.lab04-PERFORMANCEMANUAL.pdf Performance Tuning Best Practices for ESX Server 3 http://www.vmware.com/pdf/vi_performance_tuning.pdf ESX Server 3.0 Tips and Tricks - http://download3.vmware.com/vmworld/2006/tac9567.pdf Performance Monitoring and Capacity Planning http://download3.vmware.com/vmworld/2006/adc0199.pdf A Performance Comparisopn of Hypervisors http://www.vmware.com/pdf/hypervisor_performance.pdf Using esxtop to troubleshoot performance problems http://www.vmware.com/pdf/esx2_using_esxtop.pdf ESX Workload Analysis: Lessons Learned http://download3.vmware.com/vmworld/2006/adc9398.pdf 42 RTFM Guides: Upgrade Guide to ESX 3 and VirtualCenter 2 - http://www.rtfm-ed.eu/docs/vmwdocs/esx3.x-vc2upgradeguide.pdf Guide to ESX 3 Service Console - http://www.rtfm-ed.eu/docs/vmwdocs/esx3.x-vc2.xserviceconsole-guide.pdf Guide to ESX 2.x Administration I - http://www.rtfm-ed.eu/docs/vmwdocs/admin-01-esx2.x.pdf Guide to ESX 2.x Administration II - http://www.rtfm-ed.eu/docs/vmwdocs/admin-02-esx2.x.pdf Beyond the Manual - VMware ESX 2.x - http://www.rtfm-ed.eu/docs/vmwdocs/esx2.xbeyondthemanual.pdf ESX 2.x Service Console Quick Command-Line Guide - http://www.rtfmed.eu/docs/vmwdocs/esx2.x-serviceconsolecommands.pdf Guide to vCenter 1.x Administration - http://www.rtfm-ed.eu/docs/vmwdocs/admin-vcenter1.2.pdf Scripting: Simplifying Administration with Scripting APIs http://download3.vmware.com/vmworld/2006/labs2006/vmworld.06.lab12-Scripting.VI3MANUAL.pdf Security: Vmware as a secure and leak resistant sandbox environment http://www.vmware.com/community/thread.jspa?threadID=66085&tstart=0 Security Design of the Vmware Infrastructure 3 Architecture http://www.vmware.com/pdf/vi3_security_architecture_wp.pdf Vmware Infrastructure 3 Security Hardening http://www.vmware.com/pdf/vi3_security_hardening_wp.pdf VI3 Securing and Monitoring http://download3.vmware.com/vmworld/2006/labs2006/vmworld.06.lab05-SECURITY-MANUALAPPENDIX.pdf Vmware ESX Server – Providing LUN Security - http://www.vmware.com/pdf/esx_lun_security.pdf Vmware and NSA - http://www.vmware.com/company/news/releases/nsa_pr.html Service Console: Common Linux commands on the service console http://www.vmts.net/article/linuxcommand.htm Command Line Interface Tips and Tricks for Vmware ESX Server 2 and 3 http://download3.vmware.com/vmworld/2006/tac9685.pdf 43 Snapshots: Snapshots take long time to commit http://www.vmware.com/community/thread.jspa?threadID=74669 Snapshots take long time to commit http://www.vmware.com/community/thread.jspa?messageID=584324򎪄 Snapshots take long time to commit http://www.vmware.com/community/thread.jspa?threadID=73553 Beware the long snapshot - http://www.vmwarez.com/2006/11/beware-long-snapshot.html Storage: SAN Configuration Guide - http://www.vmware.com/pdf/vi3_esx_san_cfg.pdf SAN System Design and Deployment Guide http://www.vmware.com/pdf/vi3_san_design_deploy.pdf Extending a VMFS3 data store http://www.vmware.com/community/thread.jspa?threadID=65156&tstart=0 LUNS - http://www.vmware.com/community/thread.jspa?messageID=333672񑝨 HBA load balancing http://www.vmware.com/community/thread.jspa?messageID=342918񓮆 HBA Redundancy http://www.vmware.com/community/thread.jspa?messageID=428932񨮄 Configuring iSCSI in a Vmware 3 environment - http://www.vmware.com/pdf/vi3_iscsi_cfg.pdf iSCSI , NAS and IP Storage Configuration for Vmware ESX Server http://download3.vmware.com/vmworld/2006/tac9722.pdf ESX Server Raw Device Mapping - http://www.vmware.com/pdf/esx25_rawdevicemapping.pdf Unofficial list of SATA controllers that will work with ESX http://vmprofessional.com/index.php?content=sata_faq Running Vmware ESX on SATA disks - http://www.virtualization.info/2006/02/tech-runningvmware-esx-server-on-sata.html Install ESX on SATA http://www.vmware.com/community/thread.jspa?messageID=554936򇞸 Fibre channel paths http://www.vmware.com/community/thread.jspa?threadID=76763&messageID=601390#601390 http://www.vmware.com/support/esx2/doc/esx20admin_multipath_disks.html http://www.vmware.com/support/esx21/doc/esx21admin_multipath_disks.html VMFS volume symblic links/uuid - http://www.redhat.com/docs/manuals/enterprise/RHEL-3Manual/sysadmin-guide/ch-devlabel.html VMFS volume symblic links/uuid http://www.vmware.com/community/thread.jspa?messageID=606588򔅼 44 Choosing and Architecting Storage for your Environment http://download3.vmware.com/vmworld/2006/adc0135.pdf Using multi-pathing in ESX Server http://www.vmware.com/support/esx21/doc/esx21admin_multipath_disks.html Vmware Infrastructure 3, HP StorageWorks best practices http://h71019.www7.hp.com/ActiveAnswers/downloads/VMware3_StorageWorks_BestPractice.p df Time sync: Configuring windows time service (not in ESX) http://www.windowsnetworking.com/articles_tutorials/Configuring-Windows-Time-Service.html Vmware time sync and windows time service (understand windows time first) http://kb.vmware.com/kb/1318 Time sync http://www.vmware.com/community/thread.jspa?forumID=21&threadID=16115&messageID=1860 17 Timekeeping in VMware virtual machines - http://www.vmware.com/pdf/vmware_timekeeping.pdf Clock in a Linux guest runs more slowly or quickly than real time - http://kb.vmware.com/kb/1420 Troubleshooting: Troubleshooting Vmware ESX Server 3 and Vmware VirtualCenter 2 http://download3.vmware.com/vmworld/2006/mdc9694.pdf Kernrate viewer - http://www.microsoft.com/downloads/details.aspx?familyid=d6e95259-8d9d4c22-89c4-fad382eddcd1&displaylang=en VI3 Network Scenarios and Troubleshooting http://download3.vmware.com/vmworld/2006/tac9689-b.pdf VirtualCenter 2 Diagnostics - http://download3.vmware.com/vmworld/2006/tac4096.pdf Tips for Troubleshooting Vmware ESX Server Faults http://download3.vmware.com/vmworld/2006/tac0028.pdf Vm-support options http://www.vmug.nl/modules.php?name=Content&pa=showpage&pid=2&page=15 When disaster strikes Vmware - http://www-128.ibm.com/developerworks/linux/library/l-vmware/ Troubleshoot high CPU on ESX with top command - http://www1.ibm.com/support/docview.wss?rs=180&context=SSEQTP&q1=mustgather+cpu&uid=swg21132 699&loc=en_US&cs=utf-8&lang=en Upgrade from ESX 2.x to ESX 3.0: Virtual Infrastructure 3: Beta to Production - 45 http://download3.vmware.com/vmworld/2006/mdc5173.pdf Understanding Virtual Infrastructure Upgrade Moving to the new ESX 3.0/VC2.0 Release http://download3.vmware.com/vmworld/2006/tac9809.pdf Details of What's New and Improved in VMware Infrastructure 3 http://www.vmware.com/support/vi3/doc/whatsnew_esx301_vc201.html Vmware Infrastructure 3 Release Notes http://www.vmware.com/support/vi3/doc/releasenotes_esx301_vc201.html Planning a Upgrade to Vmware Infrastructure 3 http://www.vmware.com/pdf/vi3_upgrade_plan.pdf http://www.rtfm-ed.co.uk/docs/vmwdocs/ESX3.x-VC2-upgradeguide.pdf Utilities: Vmware utilities - http://petruska.stardock.net/Software/VMware.html WinScp - http://winscp.net/eng/index.php Vmount 1.1 for Vmware - http://www.abinsight.com/vm/ FastSCP - http://www.veeam.com/veeam_fast_scp.asp Virtual disk (VMDK): Repair VMDK http://www.vmware.com/community/thread.jspa?threadID=77535&messageID=606523#606523 Extend boot volume of Windows 2003 VM http://vmprofessional.com/index.php?content=2k3resize Resizing Windows VM disks - http://djberen.googlepages.com/resizingvmwarepartitions Defragging virtual disks http://www.vmware.com/community/thread.jspa?threadID=77661&tstart=0 Resizing Windows VM disks - http://www.vmprofessional.com/index.php?content=2k3resize Resizing Windows VM disks - http://h0bbel.p0ggel.org/2007/04/10/howto-expand-vm-bootpartition/ Resizing Windows VM disks - http://linux.softpedia.com/get/System/Recovery/System-RescueCD-188.shtml Unable to log on if the boot partition drive letter has changed http://support.microsoft.com/kb/249321/en-us How to restore the system/boot drive letter in Windows http://support.microsoft.com/kb/223188/en-us Virtual Center: VIrtualCenter Technical Best practices - http://www.vmware.com/pdf/vc_technical_best.pdf VirtualCenter Monitoring and Performance Statistics - 46 http://www.vmware.com/pdf/vi3_monitoring_statistics_note.pdf VirtualCenter 2.X Using Database Views - http://www.vmware.com/pdf/vc_dbviews_2x.pdf SQL Server database as a VM Guest http://www.vmware.com/community/thread.jspa?messageID=577459򌾳 SQL 2005 support - http://kb.vmware.com/kb/6565318 SQL 2005 Express edition support http://www.vmware.com/community/thread.jspa?threadID=67175 SQL 2005 support - http://www.vmware.com/community/thread.jspa?threadID=75222 Operation timeout - http://www.vmware.com/community/thread.jspa?threadID=75010 Managing remote servers http://www.vmware.com/community/thread.jspa?messageID=593730򐽂 Running VirtualCenter in a Virtual Machine - http://www.vmware.com/pdf/vi3_vc_in_vm.pdf Template Usage and Best Practices http://www.vmware.com/pdf/vc_2_templates_usage_best_practices_wp.pdf Virtual Center and remote ESX servers http://www.vmware.com/community/thread.jspa?messageID=593730򐽂 Managing Vmware VirtualCenter Roles & Permissions http://www.vmware.com/pdf/vi3_vc_roles.pdf Virtual Machine: VMware disk files explained - http://www.petri.co.il/virtual_vmware_files_explained.htm What Files Make Up a Virtual Machine? http://www.vmware.com/support/ws5/doc/ws_learning_files_in_a_vm.html Installing Windows XP on ESX http://www.vmware.com/community/thread.jspa?threadID=61099&tstart=100 SQL Server virtualization and multi-instancing http://www.microsoft.com/sql/howtobuy/virtualization.mspx How to virtualize SQL Server on Virtual server http://www.microsoft.com/technet/community/events/vs/add-62.mspx Configuring swap files on Windows 2003 VM http://www.vmware.com/community/thread.jspa?messageID=326236 Deploying Microsoft Exchange in Vmware Infrastructure http://www.vmware.com/pdf/exchange_best_practices.pdf Change vswp file location - http://www.vmware.com/community/thread.jspa?messageID=530426 Vmware Server: 10 ways to increase performance on a Vmware system http://www.petri.co.il/virtual_increase_vmware_performance.htm 47 Vmware Server Tips and Tricks - http://download3.vmware.com/vmworld/2006/dvt4696.pdf Websites: Mike Laverick's RTMF Education - http://www.rtfm-ed.co.uk Vmware Technology Center - http://www.vmware.com/vmtn/technology/ Australia and New Zealand's Premier Virtualisation Community - http://www.ozvms.com/ VM Professional - http://vmprofessional.com/ Virtual Tricks - http://virtrix.blogspot.com/ http://www.vmweekly.com/ http://x86virtualization.com/ http://sanbarrow.com/ http://www.virtualization.info/ http://www.run-virtual.com/ http://www.vmwareprofessional.com/ Vmwarez - http://www.vmwarez.com/ Virtual Strategy Magazine - http://www.virtual-strategy.com/en/home http://blog.scottlowe.org/ http://trivore.com/vmware/esx25.html Virtual Infrastructure 411 - http://www.vi411.org/ 48