Contents
Why bother?
My plan for this weekend had been to set up HA for my DHCP server given the cascading failures that resulted when I converted my DHCP server from VMware to KVM and the underlying Ceph storage filled up. Normally I’ve got a USB drive that I install from, but for whatever reason, the system that I wanted to perform the install on just sat at a blinking cursor when I tried to boot from the flash drive. I noticed that the LAN card had PXE capability, so I figured why not?
Adding PXE to the existing DHCP server
So PXE booting is going to require a TFTP server for serving the files. Most of the tutorials I’ve seen recommend installing inetd/xinetd as well to manage tftpd-hpa, but at least on Ubuntu 16.04, tftpd-hpa was immediately managed by systemd, so I didn’t see a need to install the other packages.
# apt install tftpd-hpa
This installs and starts tftpd-hpa with a server root at /var/lib/tftpboot. One thing to notice is that
# systemctl status tftpd-hpa ● tftpd-hpa.service - LSB: HPA's tftp server Loaded: loaded (/etc/init.d/tftpd-hpa; bad; vendor preset: enabled)
Seeing that “bad” in status was concerning, but it doesn’t actually affect anything. Just to verify what we’re seeing:
# systemctl is-enabled tftpd-hpa tftpd-hpa.service is not a native service, redirecting to systemd-sysv-install
It’s just that tftpd-hpa isn’t a normal systemd unit file, so it’s reporting that it’s bad. There’s nothing actually wrong, it’s just redirecting to an old-style init that doesn’t support all of the systemd options.
The only change I had to make to my existing DHCP server was to add filename "pxelinux.0"; to the existing DHCP scope.
First step is getting a bootable environment. Easiest way to do that is grab Ubuntu’s, especially since I had the ISO lying around for installs anyway. However, I’ve included the instructions to download the netboot installer and pull files from it here.
# wget http://archive.ubuntu.com/ubuntu/ubuntu/dists/xenial/main/installer-amd64/current/images/netboot/netboot.tar.gz -O ubuntu-16.04-netboot.tar.gz # mkdir ubuntu-16.04-netboot # tar zxf ubuntu-16.04-netboot.tar.gz -C ubuntu-16.04-netboot # cp -a ubuntu-16.04-netboot/ubuntu-installer /var/lib/tftpboot
Most of this wouldn’t be necessary if the PXE environment just needed to boot a single operating system. You could just throw the Ubuntu installer into /var/lib/tftpboot and off you go. However, the setup that follows creates a menu for selecting different OS options which gives me the flexibility to PXE boot different things in the future.
# cd /var/lib/tftpboot/ # mkdir boot-screens # mkdir pxelinux.cfg # cp ubuntu-installer/amd64/pxelinux.0 ./ # cp ubuntu-installer/amd64/boot-screens/ldlinux.c32 ./ # cp ubuntu-installer/amd64/boot-screens/*.c32 ./boot-screens/ # ln -s boot-screens/syslinux.cfg pxelinux.cfg/default
path boot-screens include boot-screens/menu.cfg default boot-screens/vesamenu.c32 prompt 0 timeout 100
MENU WIDTH 48
MENU MARGIN 5
MENU TABMSG
MENU TITLE Boot menu
LABEL Boot local hard drive
LOCALBOOT 0
MENU BEGIN ubuntu-16.04
MENU TITLE Ubuntu Server 16.04
LABEL mainmenu
MENU LABEL ^Back...
MENU exit
INCLUDE ubuntu-installer/amd64/boot-screens/menu.cfg
MENU END
Note that there’s a quick PXE boot menu with the default option being to boot the local hard drive. This ensures that any systems that are configured to boot from LAN with higher priority than the local drive don’t suddenly stop working because they reboot into the Ubuntu installer. A better solution would probably be to put the PXE boot subnet on a different VLAN and then change the switchport configuration after the OS has been installed, but this is sufficient for now, especially since the PXE subnet is the same as the general client subnet.
I’m not doing any Kickstart/Preseed configurations at the moment, just a manual Ubuntu install since the focus here was supposed to be HA DHCP and not setting up a PXE environment. Also, since I tend to add VMs instead of physical machines to my network, I just clone a VM template instead of doing a PXE-based install. It’s much faster and less error-prone.
Adding an HA DHCP configuration into the mix
With the PXE server up and running, I was able to boot from LAN, get Ubuntu Server 16.04 installed on the new server, and off I went.
The HA configuration in isc-dhcp-server isn’t extremely difficult. You designate one server as primary and another as secondary, add a couple different attributes, and away you go. If you’ve done any HA before, one of the major annoyances is synchronizing state. In this case, I was less concerned about the leases (since I use address reservations for anything important) and more concerned about the configuration files. Primarily, I didn’t want to have to go in and edit dhcpd.conf on two servers every time another system needed a reservation. For that reason, I decided to put my DHCP server configuration into Ansible. That’ll also make it easy to re-deploy this setup if things break.
Starting with the configuration section for failover:
{% if dhcp_server is defined and 'role' in dhcp_server %}
failover peer "{{ dhcp_server['failover'] }}" {
{{ dhcp_server['role'] }};
address {{ dhcp_server['address'] }};
port {{ dhcp_server['port'] }};
peer address {{ dhcp_server['peer'] }};
peer port {{ dhcp_server['peer_port'] }};
{%- if dhcp_server['role'] == 'primary' %}
mclt {{ dhcp_server['mclt'] }};
split {{ dhcp_server['split'] }};
{%- endif %}
max-response-delay 60;
max-unacked-updates 10;
load balance max seconds 3;
}
{% endif %}
This is a pretty basic template to define the failover peer section in dhcpd.conf. The associated host_vars are:
dhcp_server: failover: dhcp-failover role: primary address: dhcp01.geek.cm port: 519 peer: dhcp02.geek.cm peer_port: 520 mclt: 2600 split: 128
dhcp_server: failover: dhcp-failover role: secondary address: dhcp02.geek.cm port: 520 peer: dhcp01.geek.cm peer_port: 519
One problem I ran into initially was trying to set the failover peer name to different things on each system, e.g., I wanted the failover peer to be “dhcp02” on dhcp01 and “dhcp01” on dhcp02. The syslog just tells you that you have an invalid peer name which isn’t particularly helpful. Fortunately, I found this old bug report describing a change where the failover peers have to share the same name. Fixed that up and all was good.
Converting the rest of the configuration to Ansible
There were only a couple more steps to get the configuration converted to Ansible–getting the subnets defined and adding the reservations. Fortunately, they’re both pretty straightforward. I started with group_vars for the DHCP servers and defined a data structure that would suit my needs:
subnets:
- network: 192.168.0.0
netmask: 255.255.255.0
routers: 192.168.0.3
domain_name: geek.cm
dns_servers:
- 192.168.0.10
- 192.168.0.9
pxe_filename: pxelinux.0
pools:
- peer: dhcp-failover
range:
bottom: 192.168.0.100
top: 192.168.0.199
Should be pretty obvious, defines a subnet and the relevant options, adds the pxe_filename so that PXE booting is available, and specifies a DHCP range between 192.168.0.100 and 192.168.0.199. The template code that generates the config looks like:
{% for subnet in subnets %}
subnet {{ subnet['network'] }} netmask {{ subnet['netmask'] }} {
option domain-name "{{ subnet['domain_name'] }}";
option domain-name-servers {{ subnet['dns_servers']|join(',') }};
option routers {{ subnet['routers'] }};
{%- if 'pxe_filename' in subnet %}
filename "{{ subnet['pxe_filename'] }}";
{%- endif%}
{%- for pool in subnet['pools'] %}
pool {
failover peer "{{ pool['peer'] }}";
range {{ pool['range']['bottom'] }} {{ pool['range']['top'] }};
}
{%- endfor %}
}
It’s not an all-encompassing template that gives a ton of flexibility in the DHCP options, but it’s sufficient for my network at the moment. Finally, the reservations:
reservations:
- name: netgear65
mac: a1:63:92:c7:e2:34
host: netgear65.geek.cm
ip: 192.168.0.5
- name: nginx
mac: FE:2F:A0:A4:19:B4
host: nginx.geek.cm
ip: 192.168.0.11
Reservations are basically an array of hashes that have a name, a MAC address, a hostname, and an IP to reserve. And again, the template code to generate the reservations:
{% for reservation in reservations %}
host {{ reservation['name'] }} {
hardware ethernet {{ reservation['mac'] }};
server-name "{{ reservation['host'] }}";
fixed-address {{ reservation['ip'] }};
}
{% endfor %}
Finally. There it is, a high-availability DHCP setup. Now I can fill up my Ceph cluster without destroying my entire internal network.