Three Sites Federation Configuration Example
Introduction
This document presents an example of how to configure a VAE federation between 3
sites. The configuration comprises several components
1. Designating Site proxies in each site. Each site has to define a server to server as
network proxy (for communicating cross/federated site applications) and a server to
server as storage proxy (for migrating virtual machines storage). Thus, several proxies
can be defined, or the same proxy can be configured for communicating to all
federation sites. In the following example, for simplicity, we assume a single proxy
server in each site, serving both network and storage and used for communications
with all sites.
2. Select one of the sites to serve as a vpn network "main" site. This site will run
openvpn server side. All the other sites will connect as openvpn clients to the "main"
site (star architecture). See vaeproxy package: /root/vpn/vpnNewClient.sample and
/root/vpn/ vpnNewServer.sample. You need the openvpn package installed at each
site proxy.
3. Set up ssh tunnels between each 2 pairs of sites for incoming and for outgoing
streams. See vaeproxy package: /root/vpn/sshTun.sh and
/root/vpn/sshTun.remote_sample – these are ssh tunnel definitions needed for the
every pair of sites we have. You need he opnessh package installed at each site proxy.
4. Configure vae.config to reflect all the openvpn and ssh tunneling
configuration (See libvae package: /opt/reservoir/ibm/vae.config)
5.Run the port forwarder script at each site proxy: python /var/run/vaePFserver.py
see vaeproxy package : /root/vpn/vaePFserver.py
You need the python package installed at each site proxies.
This document in particular is part of the IBM libvae package (see:
/opt/reservoir/ibm/test).
All the packages mentioned are of Debian 5 and can be “apt-get” from a Debian
repository.
Federation layout
We assume three sites A, B and C. In each site a single server is serving as the proxy
(both for storage and network) to all the other sites (ProxyA, ProxyB, ProxyC). Each
Site has a unique identifier (SiteA, SiteB and SiteC)
1
Site A
openvpn Server ProxyA openvpn Server
Incoming and outgoing
Incoming and outgoing ssh tunnels
ssh tunnels
openvpn Client to ProxyA
Openvpn client to Proxy A openvpn Client to Proxy C
openvpn Client to ProxyA
Openvpn server
ProxyC ProxyB
Incoming and outgoing
ssh tunnels
Site C Site B
Network vpn channel setup
We assume Site A to be the main Site. Openvpn servers are initiated at site A.
Openvpn clients in site B and in Site C connect to the vpn servers in the main Site.
The scripts to set up the vpns:
(Note the IPs, ports and devices that are used are only an example and can be
configured differently)
In Site A, Proxy A:
openvpn --port 9000 --dev tap0 --ifconfig 192.168.200.1 255.255.255.0 --secret
static.key&
openvpn --port 9001 --dev tap1 --ifconfig 192.168.201.1 255.255.255.0 --secret
static.key&
To produce the common key run: openvpn --genkey --secret static.key
and distribute the static.key file generated to all site proxies.
Preferably this can be done at a management host and distributed
In Site B, ProxyB:
openvpn --port 9000 --remote ProxyA --dev tap0 --ifconfig 192.168.200.2
255.255.255.0 --secret static.key&
In Site C, ProxyC:
openvpn --port 9001 --remote ProxyA --dev tap1 --ifconfig 192.168.201.2
255.255.255.0 --secret static.key&
2
The above structure of 3 sites (A is main vpn site) enables connection and migration
of Virtual machines between the Main A and each of its neighbors B and C sites
(migrations of one hop distance only. Connection among the three sites) but no
migration between B and C. To achieve this last goal one need add one more vpn
directly between sites B and C. This appears in the above picture painted dark green
color where C is the server of this vpn connection (can be B as well instead). For this
new vpn we need the following scripts in addition to the above:
In Site C, ProxyC:
openvpn --port 9002 --dev tap2 --ifconfig 192.168.206.1 255.255.255.0 --secret
static.key&
In Site B, ProxyB:
openvpn --port 9002 --remote ProxyC --dev tap2 --ifconfig 192.168.206.2
255.255.255.0 --secret static.key&
The static.key can be one per all connections or unique per each vpn.
See vaeproxy package. (/root/vpn)
Ssh tunnels setup
We setup ssh tunnels for incoming and outgoing data for each pair of Sites:
(Note the ports that are used are only an example and can be configured differently
but they appear in pairs: those that are output on one site tunnel end are input to the
other site tunnel end).
In Site A:
Outgoing channel to Site B:
ssh -L2211:ProxyB:3344 ProxyB –g -4 –C N -f
Outgoing channel to Site C:
ssh –L5566:ProxyC:7788 ProxyC –g -4 –C -N -f
In Site B:
Outgoing Channel to Site A:
ssh -L1122:ProxyA:4433 ProxyA –g -4 –C -N -f
Outgoing channel to Site C:
ssh –L9911:ProxyC:8811 ProxyC –g -4 –C -N -f
In Site C:
Outgoing Channel to Site A:
ssh –L6655:ProxyA:8877 ProxyA –g -4 –C -N -f
Outgoing channel to Site B:
ssh –L1199:ProxyB:1188 ProxyB –g -4 –C -N -f
3
See vaeproxy package (/root/vpn).
Also: Run python /root/vpn/vaePFserver.py at the proxies, i.e. ProxyA, ProxyB,
ProxyC.
VAE configuration file
Vae configuration file is typical per site host members and is located in
/opt/reservoir/ibm/vae.config and should reflect the configured openvpns and ssh
tunnels of the site and also the images and master images path, the site ID of this site
and all connected sites VAN ID ranges. It is installed as prototype on any site host
member (from vaeproxy package) and should be edited to be identical for all site
members. VAN is acronym for Virtual Application Network
1. The local Site identifier should be defined after the LOCAL_SITEID keyword
LOCAL_SITEID
2. Each vpn connection to a remote site described in a line starting with NET_VPN
keyword.
The format is:
NET_VPN ip=
port= ip=
port=
is the proxy used for the openvpn setup, for communication with
the remote site. The ip-s are those defined by the script that created the openvpns
(see package vaeproxy at /root/vpn/vpnNewServer.sample or vpnNewClient.sample).
and are ports on the local and remote
proxy, that will be dedicated to listen for traffic that will be going on the VANs.
3. Each outgoing ssh tunnel is described in a line starting with
O_STORAGE_TUNNEL keyword.
The format is:
O_STORAGE_TUNNEL
is the proxy's DNS or IP.
4. Each incoming ssh tunnel is described in a line starting with
I_STORAGE_TUNNEL keyword.
The format is:
I_STORAGE_TUNNEL
4
5. The range of the VAN Ids for each site should appear in ALL vae configuration
files in all sites. For each site the vae.config file should have a dedicated line:
VANID_RANGE
For the setup defined in the previous sections, following are the vae.config examples
for each site:
vae.config for Site A:
LOCAL_SITEID SiteA
IMAGES_FS /images
MGR_COPY_MASTER no
MASTER_IMAGES_PATH /masters
ETHER_BR sw0
COW_ENABLED yes
NET_VPN SiteB ProxyA ip=192.168.200.1 port=9666 ip=192.168.200.2
port=9666
NET_VPN SiteC ProxyA ip=192.168.201.1 port=9667 ip=192.168.201.2
port=9667
O_STORAGE_TUNNEL SiteB ProxyA 2211
I_STORAGE_TUNNEL SiteB ProxyA 4433
O_STORAGE_TUNNEL SiteC ProxyA 5566
I_STORAGE_TUNNEL SiteC ProxyA 8877
VANID_RANGE SiteA 100 150
VANID_RANGE SiteB 200 250
VANID_RANGE SiteC 300 350
vae.config for Site B:
LOCAL_SITEID SiteB
IMAGES_FS /images
MGR_COPY_MASTER no
MASTER_IMAGES_PATH /masters
ETHER_BR sw0
COW_ENABLED yes
NET_VPN SiteA ProxyB ip=192.168.200.2 port=9666 ip=192.168.200.1
port=9666
O_STORAGE_TUNNEL SiteA ProxyB 1122
I_STORAGE_TUNNEL SiteA ProxyB 3344
O_STORAGE_TUNNEL SiteC ProxyB 9911
I_STORAGE_TUNNEL SiteC ProxyB 1188
VANID_RANGE SiteA 100 150
VANID_RANGE SiteB 200 250
VANID_RANGE SiteC 300 350
vae.config for Site C:
LOCAL_SITEID SiteC
IMAGES_FS /images
MGR_COPY_MASTER no
MASTER_IMAGES_PATH /masters
ETHER_BR sw0
5
COW_ENABLED yes
NET_VPN SiteA ProxyC ip=192.168.201.2 port=9667 ip=192.168.201.1
port=9667
O_STORAGE_TUNNEL SiteA ProxyC 6655
I_STORAGE_TUNNEL SiteA ProxyC 7788
O_STORAGE_TUNNEL SiteB ProxyC 1199
I_STORAGE_TUNNEL SiteB ProxyC 8811
VANID_RANGE SiteA 100 150
VANID_RANGE SiteB 200 250
VANID_RANGE SiteC 300 350
For vae.config examples see package libvae at /opt/reservoir/ibm/vae.config.sample
Note: Federated migration between different sites should always use images (storage
devices) that are NON-shared between the sites. Therefore: create
MASTER_IMAGES_PATH /masters.id files with a unique identifier for each site,
e.g. SiteA.Storage01. This symbolizes a non-shared storage although the master
image copy per site for the migrated hosts should have the same content or be the
same (one at a central web site). If each host has its copy of the master image they
should be at the same path!
In federated migration only the storage changes (cow = copy on write) that were made
at the source site compared to the master image copy are transferred to the destination
site host during migration.
6