Installing XenServer
First, a minimal OS with GPT (Debian or Ubuntu) is installed on the server. Afterwards you can log in and record the network configuration (/etc/ networking/interfaces). Important is, among other things, the gateway IP.
Download XenServer + unzip on a webserver
Download the XenServer ISO from here on its own, separate Web server: xenserver.org
wget http://downloadns.citrix.com.edgesuite.net/akdlm/6760/XenServer-6.0.201-install-cd.iso mount -o loop XenServer-6.0.201-install-cd.iso /mnt mkdir /var/www/xenserver cp -a /mnt/* /var/www/xenserver
Generate an answer file and copy it to /var/www/xenserver
Here is an example of an XML file that can be called xenserver.xml.
Caution: make sure to enter the correct IP address for the server and the gateway.
<installation mode="fresh" srtype="lvm"> <primary-disk gueststorage="yes">sda</primary-disk> <keymap>de</keymap> <hostname>xenserver-ex4s</hostname> <root-password>my_password</root-password> <source type ="url">http://xx.xx.xx.xx/xenserver/</source> <!-- No Post install scripts configured --> <admin-interface name="eth0" proto="static"> <ip>Hetzner Server IP</ip> <subnet-mask>255.255.255.224</subnet-mask> <gateway>Hetzner Gateway IP</gateway> </admin-interface> <nameserver>213.133.98.98</nameserver> <nameserver>213.133.99.99</nameserver> <nameserver>213.133.100.100</nameserver> <timezone>Europe/Berlin</timezone> <time-config-method>ntp</time-config-method> <ntp-servers>ntp</ntp-servers> <ntpservers>213.239.239.164</ntpservers> <ntpservers>213.239.239.165</ntpservers> <ntpservers>213.239.239.166</ntpservers> </installation>
This file should be stored in the same directory as the rest of the home directory of the XenServer CD.
Adjustments for PXE boot
On the new server: copy the following files from your own web server to the server intended for the XenServer installation:
cd /boot wget http://www.example.com/xenserver/install.img wget http://www.example.com/xenserver/boot/vmlinuz wget http://www.example.com/xenserver/boot/xen.gz
Now you need to customize the Bootloader configuration on the new server. With the minimal version of Ubuntu 12.04 GRUB2 is used. In the /boot/grub/grub.cfg file the first entry needs to be changed (see the last 3 lines):
if [ "${linux_gfx_mode}" != "text" ]; then load_video; fi
menuentry 'Ubuntu, with Linux 3.2.0-24-generic' --class ubuntu --class gnu-linux --class gnu --class os {
recordfail
gfxmode $linux_gfx_mode
insmod gzio
insmod raid
insmod mdraid1x
insmod part_gpt
insmod part_gpt
insmod ext2
set root='(mduuid/xxxxxxxxxxxxx)'
search --no-floppy --fs-uuid --set=root 2caba589-8a66-457a-81a2-2f43c7203736
multiboot /xen.gz dom0_mem=752M acpi=off nosmp noapic noirqbalance
module /vmlinuz answerfile=http://<IP of the remote server>/xenserver/xenserver.xml install
module /install.img
}
Note: The URL of the answer file must be specified using the IP of the server, not the host name. Therefore, be careful when you use virtual hosts, etc.
Reboot
Restart the server via a reboot. Now the XenServer installation should start, something you can check on the external web server via “tail -f /var/log/apache2/access.log”.
If, even after a few minutes, there is no activity, the server is probably stuck in the Boot menu. In this case, simply request a KVM (LARA) and confirm the first entry with <ENTER>.
Software RAID1
After the instalaltion the XenServer should be accessible via SSH with the password entered in the answer file. XenServer 6 uses GPT instead of MBR to set up the partitions. The 3 TB drives in an EX4 etc. are thus fully used.
Setting up the second drive sdb
/dev/sda should now contain 3 partitions. To transfer the partitions evenly you can look at the partitions on /dev/sda:
sgdisk -p /dev/sda
Delete the partitions on /dev/sdb:
sgdisk --zap-all /dev/sdb
Create a new GPT table:
sgdisk --mbrtogpt --clear /dev/sdb
Add the start and end of the respective partitions with the following commands (here taken from a 3TB HDD):
sgdisk --new=1:2048:8388641 /dev/sdb sgdisk --typecode=1:fd00 /dev/sdb sgdisk --new=2:8390656:16777249 /dev/sdb sgdisk --typecode=2:fd00 /dev/sdb sgdisk --new=3:16779264:5860533134 /dev/sdb sgdisk --typecode=3:fd00 /dev/sdb sgdisk /dev/sda --attributes=1:set:2 sgdisk /dev/sdb --attributes=1:set:2 sgdisk --typecode=1:fd00 /dev/sda sgdisk --typecode=2:fd00 /dev/sda sgdisk --typecode=3:fd00 /dev/sda
Creating a RAID
The RAID1 setup by Hetzner is still partially in place. XenServer has configured 3 partitions on /dev/sda and left /dev/sdb intact, as shown by the answer file in the example. The remains of the RAID are removed:
mdadm --stop /dev/md0
Create a new RAID:
mknod /dev/md0 b 9 0 mknod /dev/md1 b 9 1 mknod /dev/md2 b 9 2 mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb1 mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2 mdadm --create /dev/md2 --level=1 --raid-devices=2 missing /dev/sdb3
Optional: Creating bitmaps for each RAID device. Bitmaps have little influence on the performance but significantly reduce the time in which a failed array must be re-synchronized.
mdadm --grow /dev/md0 -b internal mdadm --grow /dev/md1 -b internal mdadm --grow /dev/md2 -b internal
Save the new RAID configuration in an updated mdadm.conf file:
mdadm --examine --scan > /etc/mdadm.conf
Copy Store Manager Data to RAID:
pvcreate -ff /dev/md2 vgextend VG_<TAB> /dev/md2 pvmove /dev/sda3 /dev/md2
Remove /dev/sda3 from the SR volume group:
vgreduce VG_<TAB> /dev/sda3 pvremove /dev/sda3
Mount /dev/md0 and copy the file system:
mkfs.ext3 /dev/md0 mount /dev/md0 /mnt cd / cp -axv . /mnt
Now the file /mnt/etc/fstab needs to be adjusted:
Note: the editor nano is not recommended, as it inserts an automatic line break with lines that are long, and this can cause the configuration file to become unreadable. This behavior can be bypassed by using “nano –nowrap”, or by simply using another editor, such as vi.
vi /mnt/etc/fstab
Replace the parameter “root=LABEL=root-xxxxxx” with “/dev/md0”
Create a new Initrd:
mkdir /mnt/root/initrd-raid mkinitrd -v --fstab=/mnt/etc/fstab /mnt/root/initrd-raid/initrd-`uname -r`-raid.img `uname -r` cd /mnt/root/initrd-raid zcat initrd-`uname -r`-raid.img | cpio -i
Edit the file “init” and add the following for “raidautorun /dev/md0”:
raidautorun /dev/md1 raidautorun /dev/md2
Copy the new ramdisk to /mnt/boot:
find . -print | cpio -o -Hnewc | gzip -c > /mnt/boot/initrd-`uname -r`-raid.img rm /mnt/boot/initrd-2.6-xen.img ln -s initrd-`uname -r`-raid.img /mnt/boot/initrd-2.6-xen.img vi /mnt/boot/extlinux.conf
Now replace “root=LABEL=root-xyz” with “root=/dev/md0” in all the menu entries.
Setup the MBR for GPT on /dev/sdb:
cat /mnt/usr/share/syslinux/gptmbr.bin > /dev/sdb cd /mnt extlinux --raid -i boot/ umount /dev/md0 sync
The RAID array is now almost complete, it is only missing /dev/sda1. Reboot into the Rescue System (activate the Rescue System in Robot, copy the password, reboot the server via a hardware reset). Run the following in the Rescue System:
mdadm -a /dev/md0 /dev/sda1 mdadm -a /dev/md2 /dev/sda3
The RAID array must now be synchronized for the first time, which can take a moment(!) (Based on this guide on a freshly installed EX4S about 6 hours). Progress can be observed as follows:
watch -n 1 cat /proc/mdstat
Embed a local ISO Library
XenServer has no possibility to offer ISO images via local storage. To enable this nonetheless, in the data area of the LVM a new LV can be created and passed through via a local NFS server. This is not optimal, but it serves its purpose. Source:http://forums.citrix.com/thread.jspa?messageID=1393861&tstart=0
Find your own VG
vgscan #Reading all physical volumes. This may take a while... #Found volume group "VG_XenStorage-709d46ed-8193-d470-4ab8-21953af4f863" using metadata type lvm2
Create a new LVM
(Example with 20 GB)
lvcreate -L 20G -n ISO VG_XenStorage-<myID> #Logical volume "ISO" created
Creating a file system
mkfs.ext3 /dev/VG_XenStorage-<myID>/ISO
Adjust /etc/exports
mkdir /ISO echo "/ISO 127.0.0.1(rw,no_root_squash,sync)" >> /etc/exports
Activate NFS and Portmap
chkconfig --level 345 nfs on chkconfig --level 345 portmap on service nfs start service portmap start
Mount at boot
Add the following to the end of the file “/etc/rc.local”:
lvchange -a y /dev/VG_XenStorage-<myID>/ISO mount /dev/VG_XenStorage-<myID>/ISO /ISO
Enter the ISO Libray in XenCenter
In XenCenter, you can now add a new storage library of the type “NFS ISO”. As a mount point “localhost:/ISO” can be specified.
Network Configuration
Configure the Host as a Router
The XenServer can be configured as a router via changes in the file “/etc/sysctl.conf” (the first few lines up to and including “net.ipv4.ip_forward = 0” must be replaced with the following):
# Kernel sysctl configuration file for Red Hat Linux # # For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and # sysctl.conf(5) for more details. # Controls IP packet forwarding net.ipv4.ip_forward = 1 net.ipv6.conf.all.forwarding=1 # Controls proxy arp net.ipv4.conf.default.proxy_arp = 0 # Turn off redirects net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.lo.send_redirects = 0 net.ipv4.conf.xenbr0.send_redirects = 0
The settings are now active after every reboot. You can also apply them without restarting directly via the sysctl -p command:
sysctl -p
IPv4
Configuration of the additional IP addresses
Each dedicated server at Hetzner comes with a single IPv4 address. However, you can order up to 3 additional single IPv4 addresses. See: IP Addresses
Configuration of a failover IP address
A failover IP address is an additional address that can be switched to any other dedicated Hetzner server.
Configuration of an additional subnet
In the standard configuration with the 3 additional IPv4 addresses you can run up to 3 virtual machines. You can also order subnets. See: IP Addresses
For an additional subnet the same applies as above. An IP address of the subnet acts as a gateway and the others can be used for guests.
To make sure the host knows that it should route packets from that subnet, we need to add the first IP address of the subnet to the xenbr0 interface (example using xx.yy.177.160/27):
ip addr add xx.yy.177.161/27 dev xenbr0
Routing to adjacent servers
If you have ordered multiple servers simultaneously and they are located in the same rack, and have consecutive IP addresses, you will find that you cannot ping or SSH into those servers. This is because, while normally all servers on the same subnet with no gateway should be reachable, at Hetzner this is disabled for security reasons. The neighboring servers are accessible only via Hetzner’s gateway. The routing configuration is as follows:
route add -net xx.yy.44.64 netmask 255.255.255.192 gw xx.yy.44.65 xenbr0
In this case xx.yy.44.65 is the IP address of the Hetzner gateway. To obtain the IP address of the entire subnet, we simply subtract 1, which gives us xx.yy.44.64.
These two commands must be run after every host reboot, otherwise the virtual machines have no connection to the internet.
Configuration example:
- The server is allocated the following IP addresses: xx.yy.44.76, xx.yy.44.105, xx.yy.44.108 and xx.yy.44.110.
- These addresses are all in the same subnet xx.yy.44.64/26.
- The gateway of this subnet is xx.yy.44.65.
- The XenServer virtual machines get the IPs xx.yy.44.105, xx.yy.44.108 and xx.yy.44.110
- The gateway is xx.yy.44.76.
- The additional ordered subnet with 32 IPs is xx.yy.177.160/27
- The first and last IP address are not usable, since they are the subnet address and the broadcast address
- The IP address xx.yy.177.161 acts as gateway, so there are 29 IP addresses available: xx.yy.177.162 – xx.yy.177.191
- The gateway for the virtual machines in this subnet is xx.yy.177.161.
The following can be entered into the file “/etc/sysconfig/networking-scripts/ifcfg-xenbr0” to make sure the entries are available even after a reboot:
up ip addr add xx.yy.177.161/27 dev xenbr0 down ip addr del xx.yy.177.161/27 dev xenbr0 up route add -net xx.yy.44.64 netmask 255.255.255.192 gw xx.yy.44.65 xenbr0 down route del -net xx.yy.44.64 netmask 255.255.255.192 gw xx.yy.44.65 xenbr0
IPv6
All servers come with a /64 IPv6 subnet. For clients who received their servers before February 2013 this subnet can be ordered (freely) via Robot, and will be automatically activated.
An example subnet would be 2a01:4f8:xxx:xxxx::/64
In this example the XenServer would get the IP 2a01:4f8:161:xxxx::2/112.
Since XenServer does not support IPv6 configurations via the management interface in the xconsole, you will need to use a small script. This is located in “/etc/init.d” and is called “network6”. The script looks like this:
#!/bin/bash
#
# network6 This starts and stops ipv6 on xenbr0
#
# chkconfig: 2345 15 85
# description: ipv6 configuration
# source: http://www.wirrewelt.de/entry/ipv6-mit-xenserver-5-5-bei-hetzner
# Copyright 2010 Sascha Huck
#
### BEGIN INIT INFO
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
### END INIT INFO
# Source function library.
. /etc/rc.d/init.d/functions
# Von Hetzner zugewiesenes Gateway
IPV6GATEWAY=fe80::1
# Eigene Adresse und eigenes Subnetz
IPV6HOSTADDR=2a01:4f8:161:xxxx::2/112
IPV6HOSTSUBNET=2a01:4f8:161:xxxx::0/112
usage ()
{
echo "Usage: service $prog {start|stop}"
RETVAL=1
}
start ()
{
# Insert kernel modules
modprobe esp6
# Configure our ip address
ip -6 addr add $IPV6HOSTADDR dev xenbr0
# Configure routing to hetzner gateway
ip -6 route add $IPV6GATEWAY dev eth0
ip -6 route add $IPV6GATEWAY dev xenbr0
ip -6 route add default via $IPV6GATEWAY
# Enable ipv6 forwarding (don't work in /etc/sysctl.conf)
sysctl -w net.ipv6.conf.all.forwarding=1
}
stop ()
{
}
case "$1" in
start) start; RETVAL=$? ;;
stop) stop; RETVAL=$? ;;
*) usage ; RETVAL=2 ;;
esac
exit $RETVAL
########
NOTE: various entries need to be adjusted according to the configuration you have. Make sure to appropriately edit the script before using it. With
chmod a+x
the script is made executable and with
chkconfig network6 on
it is added to the boot process.
Loading the kernel module esp6 is a workaround for XenServer, since “modprobe ipv6” is acknowledged with “Module no not found”, while “modprobe esp6” loads the IPv6 module.
In the VMs (if they run on Linux) you can add the following settings:
ip addr add 2a01:4f8:161:xxx::y/64 dev eth0 ip route add default via 2a01:4f8:161:xxx::2
Change the driver for the network card (optional)
The default driver in XenServer that gets loaded for the Realtek network card (r8169) produces packet loss under some circumstances. Therefore, it can be replaced with a different driver: r8168. To compile the driver the appropriate Driver Development Kit (DDK) is required, which can be downloaded from the Citrix website:http://support.citrix.com/article/CTX138885 (for XenServer 6.2.0 with Hotfix XS62E004)
Assumed is that the ISO file has already been copied to /ISO. Firstly the VM Storage should be marked as the default via XenCenter (right click -> Default Storage) as otherwise the xe vm-import command doesn’t work.
Mount the ISO:
mkdir /mnt/iso mount -o loop /local/iso/XenServer-6.2.0-XS62E004-ddk.iso /mnt/iso
Import DDK VM:
xe vm-import filename=/mnt/iso/ddk/ova.xml
In XenCenter a network device needs to be added to the virtual machine. After that the machine can be restarted and a root password can be set with which you can log in.
Now we can add the correct network settings for the VM in the following files: “/etc/sysconfig/network-scripts/ifcfg-eth0”
DEVICE=eth0 BOOTPROTO=static IPADDR=<IP address of the VM> NETMASK=255.255.255.224 ONBOOT=yes TYPE=ethernet
“/etc/sysconfig/network-scripts/route-eth0”
IP-ADRESSE-DER_VM dev eth0 scope link default via IP-DES-XENSERVERS-BZW-BEI-IP-NETZ-DIE-DES-xenbr0:1
“/etc/resolv.conf”
nameserver 213.133.99.99 nameserver 213.133.100.100
Now grab the driver, unzip it and compile it:
cd /root wget http://r8168.googlecode.com/files/r8168-8.037.00.tar.bz2 tar xjf r8168-8.037.00.tar.bz2 cd r8168-8.037.00 make all
The new driver (src/r8168.ko) needs to be transferred to the host system: “/lib/modules/<newest kernel version>/kernel/drivers/net/”
Give the new driver the proper permissions:
chmod 0744 /lib/modules/<newest kernel version>/kernel/drivers/net/r8168.ko
Now add the new driver to the “/etc/modprobe.conf” file:
echo "alias eth0 r8168" > /etc/modprobe.conf
The compiled driver must now be activated. Since the server will not be reachable via the network during this time (a few seconds) the necessary instructions will all be added to a single command:
rmmod r8169 && depmod -a && modprobe r8168 && service network restart && service ipaliases restart
After a few seconds the server should be reachable via the network again with the new driver.
lspci -nnk | grep -i net -A2
The last line should read: “Kernel driver in use: r8168”
Transferring virtual machines from another (older) Xenserver
XenServer is fortunately quite flexible in terms of transferring other virtual machines on a XenServer host. There are many ways to do this, one of which is to remotely mount a directory (as NFS) of the new XenServer host from the old XenServer host. In this directory a VM is exported to the other host (via “xe vm-export uuid=xxx-xxx filename=vm1.xva”) and then imported on the new host (“xe vm-import filename=vm1.xva”).
