Using kickstart to automate ESX deployments: Provisioning an ESX server with kickstart can take deployment time from hours to just a few minutes. Using a CD for deployment and not accounting for the time it takes to boot, my typical installation will last 6.5 minutes. In this article I will break down each section of the kickstart and provide an example of each option. At the end of the article I will provide a downloadable kickstart file that you can modify to fit your own specific needs. In Part II: Creating a custom boot CD, I'll show you how to take one or more kickstart files and slipstream them into a single CD.
General Kickstart Options
install # Localization lang en_US keyboard us # Mouse Configuration mouse generic3ps/2 --device psaux # Skip X configuration skipx # Text install text # Network information network --device eth0 --bootproto static --ip 172.16.1.10 --netmask 255.255.255.0 --gateway 172.16.1.1 \ --nameserver 172.16.1.2 --hostname esx01.yourdomain.com # Encrypted root password rootpw imaweenie # Disable the firewall firewall --disabled # Auth Configuration authconfig --enableshadow --enablemd5 # Timezone timezone America/Los_Angeles # Bootloader config bootloader --useLilo --location=mbr # Reboot reboot
VMware Licensing
There are 2 options that can be set here. First, the "vmaccepteula" can be specified to acknowledge that you have read and understand the EULA. Secondly serial number for both ESX and vSMP can be specified.
# VMware Licensing vmaccepteula vmserialnum --esx=XXXXX-XXXXX-XXXXX-XXXXX --esxsmp=XXXXX-XXXXX-XXXXX-XXXXX
Console Memory
The option "vmservconmem" can be specified to allocate memory to the service console. 192 for up to 8 virtual machines, 272 for up to 16 virtual machines, 384 for up to 32 virtual machines, 512 for over 32 virtual machines, or 800 if you want to assign the maximum amount of memory to the console.
# Amount of memory to reserve for the console OS vmservconmem --reserved=512
Partitioning
Here is an example of how I set up my partition table. If you use Compaq/HP hardware as I do, you will need to prepend the "ccis" driver and use the non-standard disk naming convention. If you don't use Compaq/HP hardware you will most likely just need to use "sda" or whatever disk it is you want to partition. I like to place the vmkcore partition at the end of the drive so that in the unlikely event that a coredump writes beyond 100M it will not "bleed" into any of my other paritions.
# Partitioning # *Note: ESX 3.0 Will require a 100M clearpart --all --initlabel part /boot --size 100 --ondisk part / --size 10240 --ondisk part swap --size 2048 --ondisk part /vmimages --size 10240 --ondisk part local --size 1 --ondisk part vmkcore --size 100 --ondisk
/boot parition for an upgrade. cciss/c0d0 cciss/c0d0 cciss/c0d0 cciss/c0d0 cciss/c0d0 cciss/c0d0 --fstype --fstype --fstype --fstype --fstype --fstype ext3 ext3 swap ext3 vmfs2 vmkcore --asprimary --asprimary --asprimary --grow
VMKswap
Here an 8.0G VMkernel swapfile is generated, and placed on the partition that is labeled "local" in the step above. The swap file name can be changed, but SwapFile.vswp is the standard.
# VMKernel Swap vmswap --volume="local" --size="8192" --name "SwapFile.vswp"
PCI Devices
For this step you will need to gather data from /etc/vmware/devnames.conf about the PCI devices in your system. For my test system I have 2 onboard GigE ports, an Intel 1000MT quad port GigE card, and a single Qlogic 2340 HBA. The first NIC and the local storage controller are set to "shared" all other devices are assigned exclusively to the virtual machines. Examining /etc/vmware/devnames.conf gave me all the following pci information which is shown in bus/slot/function notation. [root@esx01 vmware]# more devnames.conf 002:04.0 scsi vmhba0 003:06.0 nic vmnic0 003:06.1 nic vmnic1 006:04.0 nic vmnic2 006:04.1 nic vmnic3 006:06.0 nic vmnic4 006:06.1 nic vmnic5 007:09.0 fc vmhba1
# Assign all PCI devices ( All of these device IDs can be obtained by looking at /etc/vmware/hwconfig ) # 2/4/0 scsi = vmhba0 (shared) Onboard RAID controller # 3/6/0 nic = vmnic0 (shared) First onboard GigE NIC # 3/6/1 nic = vmnic1 (vm) Second onboard GigE NIC # 6/4/0 nic = vmnic2 (vm) Intel 1000MT NIC Port 1 # 6/4/1 nic = vmnic3 (vm) Intel 1000MT NIC Port 2 # 6/6/0 nic = vmnic4 (vm) Intel 1000MT NIC Port 3 # 6/6/1 nic = vmnic5 (vm) Intel 1000MT NIC Port 4 # 7/9/0 fc = vmhba1 (vm) Qlogic 2340 Fibre HBA vmpcidivy --shared=2/4/0 --shared=3/6/0 --vms=3/6/1 --vms=6/4/0 --vms=6/4/1 --vms=6/6/0 --vms=6/6/1 --vms=7/9/0
Virtual Switches
Below I have 3 virtual switches: "vmotion", "dmz1" and "dmz2" which each have a single vmnic assigned to them. I have also created a bond named "internal" which has 2 vmnics assigned, as well as port groups for 5 seperate VLANS. An internal only vSwitch (vmxnet) named "private_network" has been created and no vmnic have been assigned to it.
# Set up virtual switches. vmnetswitch --name="vmotion" vmnetswitch --name="internal" vmnetswitch --name="vlan_1" vmnetswitch --name="vlan_2" vmnetswitch --name="vlan_3" vmnetswitch --name="vlan_4"
--vmnic=vmnic0 --vmnic=vmnic1 --vmnic=vmnic2 --vmnic="internal.1" --vmnic="internal.2" --vmnic="internal.3" --vmnic="internal.4"
vmnetswitch vmnetswitch vmnetswitch vmnetswitch
--name="vlan_5" --vmnic="internal.5" --name="dmz1" --vmnic=vmnic4 --name="dmz2" --vmnic=vmnic5 --name="private_network"
%post configuration
You can do most anything in the %post section of your kickstart, and most of it will be specific to your environment. One piece that nearly everyone needs is to download and apply the latest patch to the installation which can be done like so: (note, this section still under development ). Another general modification is to set up /etc/resolv.conf for your secondary ( or tertiary ) nameservers and list any domains that you want to search first for name resolution information.
# Modify /etc/resolv.conf cat > /etc/resolv.conf << EOF search yourdomain.com nameserver 172.16.1.2 nameserver 172.16.1.3 EOF
Configure NTP ( replace 172.16.1.4 with the ip address or FQDN of your timeserver ).
# NTP Configuration chkconfig --level 345 ntpd on perl -spi -e 's|# restrict mytrustedtimeserverip mask 255.255.255.255 nomodify notrap noquery|\ restrict 172.16.1.4 mask 255.255.255.255 nomodify notrap noquery|' /etc/ntp.conf perl -spi -e 's|# server mytrustedtimeserverip|server 172.16.1.4|' /etc/ntp.conf cat > /etc/ntp/step-tickers << EOF 172.16.1.4 EOF
Install vmkusage
# Install vmkusage /usr/bin/vmkusage -regroove cat > /etc/cron.d/vmkusage-cron.sh << EOF #!/bin/bash */1 * * * * root /usr/bin/vmkusage > /dev/null 2>&1 EOF /bin/chmod +x /etc/cron.d/vmkusage-cron.sh
I'm forced by policy to put a banner in place to scare away the hackers.
# Set up restriction banners perl -spi -e 's|#Banner /some/path|Banner /etc/restricted_access|' /etc/ssh/sshd_config cat > /etc/restricted_access << EOF WARNING: U R NOT 3L33t, b3w4r3 of h4X0r! EOF
ks.cfg
install # Localization lang en_US keyboard us # Mouse Configuration mouse generic3ps/2 --device psaux # Skip X configuration skipx
# Text install text # Network information network --device eth0 --bootproto static --ip 172.16.1.10 --netmask 255.255.255.0 -gateway 172.16.1.1 --nameserver 172.16.1.2 --hostname esx01.yourdomain.com # Encrypted root password rootpw --imaweenie # Disable the firewall firewall --disabled # Auth Configuration authconfig --enableshadow --enablemd5 # Timezone timezone America/Los_Angeles # Bootloader config bootloader --useLilo --location=mbr # Reboot reboot # VMware Licensing vmaccepteula vmserialnum --esx=XXXXX-XXXXX-XXXXX-XXXXX --esxsmp=XXXXX-XXXXX-XXXXX-XXXXX # Amount of memory to reserve for the console OS # 192M up to 8 virtual machines # 272M up to 16 virtual machines # 384M up to 32 virtual machines # 512 more than 32 virtual machines vmservconmem --reserved=512 # Partitioning # *Note: ESX 3.0 Will require a 100M clearpart --all --initlabel part /boot --size 100 --ondisk part / --size 10240 --ondisk part swap --size 2048 --ondisk part /vmimages --size 10240 --ondisk part local --size 1 --ondisk part vmkcore --size 100 --ondisk /boot parition for an upgrade. cciss/c0d0 cciss/c0d0 cciss/c0d0 cciss/c0d0 cciss/c0d0 cciss/c0d0 --fstype --fstype --fstype --fstype --fstype --fstype ext3 ext3 swap ext3 vmfs2 vmkcore --asprimary --asprimary --asprimary --grow
# Not sure how to specify this without device labels vmswap --volume="local" --size="8192" --name "SwapFile.vswp" # Set up virtual switches. vmnetswitch --name="vmotion" vmnetswitch --name="internal" vmnetswitch --name="vlan_1" vmnetswitch --name="vlan_2" vmnetswitch --name="vlan_3" vmnetswitch --name="vlan_4" vmnetswitch --name="vlan_5" vmnetswitch --name="dmz1" vmnetswitch --name="dmz2" --vmnic=vmnic0 --vmnic=vmnic1 --vmnic=vmnic2 --vmnic="internal.1" --vmnic="internal.2" --vmnic="internal.3" --vmnic="internal.4" --vmnic="internal.5" --vmnic=vmnic4 --vmnic=vmnic5
vmnetswitch --name="private_network" # Assign all PCI devices ( All of these device IDs can be obtained by looking at /etc/vmware/hwconfig ) # 2/4/0 scsi = vmhba0 (shared) Onboard RAID controller # 3/6/0 nic = vmnic0 (shared) First onboard GigE NIC # 3/6/1 nic = vmnic1 (vm) Second onboard GigE NIC # 6/4/0 nic = vmnic2 (vm) Intel 1000MT NIC Port 1 # 6/4/1 nic = vmnic3 (vm) Intel 1000MT NIC Port 2 # 6/6/0 nic = vmnic4 (vm) Intel 1000MT NIC Port 3 # 6/6/1 nic = vmnic5 (vm) Intel 1000MT NIC Port 4 # 7/9/0 fc = vmhba1 (vm) Qlogic 2340 Fibre HBA vmpcidivy --shared=2/4/0 --shared=3/6/0 --vms=3/6/1 --vms=6/4/0 --vms=6/4/1 --vms=6/6/0 -vms=6/6/1 --vms=7/9/0 %packages @ ESX Server kernel-smp %post # Modify /etc/resolv.conf cat > /etc/resolv.conf << EOF search yourdomain.com nameserver 172.16.1.2 nameserver 172.16.1.3 EOF # NTP Configuration chkconfig --level 345 ntpd on perl -spi -e 's|# restrict mytrustedtimeserverip mask 255.255.255.255 nomodify notrap noquery|restrict 172.16.1.4 mask 255.255.255.255 nomodify notrap noquery|' /etc/ntp.conf perl -spi -e 's|# server mytrustedtimeserverip|server 172.16.1.4|' /etc/ntp.conf cat > /etc/ntp/step-tickers << EOF 172.16.1.4 EOF # Install vmkusage /usr/bin/vmkusage -regroove cat < /etc/cron.d/vmkusage-cron.sh #!/bin/bash */1 * * * * root /usr/bin/vmkusage > /dev/null 2>&1 EOF /bin/chmod +x /etc/cron.d/vmkusage-cron.sh # Set up restriction banners perl -spi -e 's|#Banner /some/path|Banner /etc/restricted_access|' /etc/ssh/sshd_config cat > /etc/restricted_access << EOF WARNING: U R NOT 3L33t, b3w4r3 of h4X0r! EOF
Create kickstart configuration files for each machine
See Kickstart Part I for more details.
Generate a new boot image for the installation CD
The boot image for the ESX installation CD can be found in the isolinux directory at the root of the cd. We will first need to copy the initrd.img to a location where we can modify it.
[root@esx01 isolinux]# cp initrd.img /root
Next, we need to extract the image, and mount it as a loopback device.
[root@esx01 ~]# gunzip initrd.img --suffix .img [root@esx01 ~]# mount -o loop /root/initrd /mnt/floppy/
Copy all of the kickstart configuration files to /mnt/floppy
[root@esx01 ~]# [root@esx01 ~]# bin esx01.cfg dev esx02.cfg
cp esx0[1-5].cfg /mnt/floppy/ ls /mnt/floppy/ esx03.cfg esx05.cfg linuxrc esx04.cfg etc lost+found
modules proc
sbin tmp
var
Finally, unmount and recompress the initrd
[root@esx01 ~]# umount /mnt/floppy/ [root@esx01 ~]# gzip --suffix .img initrd
Modify isolinux.cfg
Modify isolinux.cfg to create entries for each of your configuration files as shown.
default esx prompt 1 timeout 200 display boot.msg F1 boot.msg label esx kernel vmlinuz append apic devfs=nomount ramdisk_size=7168 vga=788 initrd=initrd.img label esx01 kernel vmlinuz append vmware apic text ks=file:/esx01.cfg ramdisk_size=7168 initrd=initrd.img label esx02 kernel vmlinuz append vmware apic text ks=file:/esx02.cfg ramdisk_size=7168 initrd=initrd.img label esx03 kernel vmlinuz append vmware apic text ks=file:/esx03.cfg ramdisk_size=7168 initrd=initrd.img label esx04 kernel vmlinuz append vmware apic text ks=file:/esx04.cfg ramdisk_size=7168 initrd=initrd.img label esx05 kernel vmlinuz append vmware apic text ks=file:/esx05.cfg ramdisk_size=7168 initrd=initrd.img
local local local local local
Inserting initrd.img and isolinux.cfg back into a bootable cdrom
Unfortunately, ESX does not come with the tools necessary to create an iso image from a directory structure, so for this step you will need another machine. For the linux section, I will assume that you have the mkisofs package available to you.
Creating an ISO image in linux
Copy the entire source of the ESX cd to /kickstart
[root@esx01 ~]# cp -pr /mnt/cdrom/.* /kickstart/
esx 3.0.x only, copy .discinfo from /mnt/cdrom to /kickstart
[root@esx01 ~]# cp -p /mnt/cdrom/.discinfo /kickstart/
Copy the modified initrd.img and isolinux.cfg to /kickstart/isolinux directory
[root@esx01 ~]# cp {isolinux.cfg,initrd.img} /kickstart/isolinux/ cp: overwrite `/kickstart/isolinux/isolinux.cfg'? y cp: overwrite `/kickstart/isolinux/initrd.img'? y
You may also want to edit the file /kickstart/isolinux/boot.msg to reflect the new boot entries that you added to isolinux.cfg.
Generate a new ISO image from the /kickstart directory
[root@esx01 kickstart]# mkisofs -l -J -R -r -T -o /esx-custom.iso -b isolinux/isolinux.bin -c \ isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table /kickstart
You will now have an ISO image located at /esx-custom.iso, burn this to a cd with whatever tool you have available.