Migrating My Servers (and me!) to NYC

I just moved to New York City! Moving, as you probably know, is a lot of work, and moving to another country is even trickier. On top of this, I have a bunch of servers in my house, which I wanted to bring with me. I just finished migrating them with minimal downtime, and I wanted to write a bit about how I did this.

I've had a homelab in some form since I was about 12 years old. I started with used desktops and gradually moved to off-lease rackmount equipment. I like rackmount servers because they're dirt cheap, extremely reliable, and I can throw a lot of them into a rack without taking up too much space.

Today, my servers host all sorts of important things, like my email, my git server, and this website. In general, I try to self-host anything I can. This can be really freeing - I'm not beholden to external services - but it can also make me feel tethered. I've never had a server go down when I was on vacation, but it's always something I'm worried about when travelling.

Everything important is virtualized and runs on top of a single server running the Proxmox hypervisor. I have a few other servers, but they do less-important things and their uptime is far less critical.

As part of my move to NYC, I decided I wanted to colocate the main server.

Colocation is the practice of renting out physical space in a datacenter and supplying your own hardware. This is a very economic way to get a large amount of compute in a datacenter for relatively cheap. Furthermore, you retain ownership of the hardware. I'm a lot more comfortable with this than I am with the Cloud.

There's a ton of colocation available in NYC and surrounding areas. I'm paying a little under $100 USD/month for 1u of rack space and a symmetric gigabit connection nearby in New Jersey. In that space I managed to cram almost a terabyte of RAM and over 10TB of high-speed flash storage. From the Cloud, this would likely cost thousands per month. Colos are also great because they offer remote hands - you can pay an hourly rate to have someone at the datacenter physically manage your server in some way - e.g. to upgrade the RAM or to replace a failed drive. Of course, I can also physically go to the datacenter and perform that work myself. No longer am I tethered to my servers, and I can be much more at ease when travelling.

My goal is pretty straightforward. I want to migrate my primary server from my house in Eastern Canada to a datacenter in NJ, and I want to do it with minimal downtime. I considered just driving it down, but it's a 12 hour drive, and that's more downtime than I was willing to deal with.

Instead, I would have to do something a bit more involved.

At a high level, I decided to do the following:

• Set up a second Proxmox node (Node B) and cluster it with the primary hypervisor (Node A)

• Migrate all of the critical VMs from Node A to Node B

• Shut down Node A and ship it to NJ

• Set up a tunnel between the two nodes and re-cluster them

• Migrate the remaining VMs over the tunnel from Node B to Node A

• Destroy the cluster

• Shut down Node B

Proxmox supports live migrations between nodes in a cluster, meaning the VMs could be moved while running. This allowed me to reduce downtime from many hours to just a minute or two.

Node B was very nicely donated to me from my friend Olivia. I didn't have enough spare compute otherwise so this was greatly appreciated.

Clustering the nodes and migrating the VMs was straightforward; it's something I've done many times in the past. Shipping was a bit more annoying. I've been on the receiving end of many server deliveries but I've never packaged one myself. Turns out it's a lot of work! Servers are awkward and heavy and they need a lot of packing material. I am even more grateful to Olivia now, knowing she had to go through the same thing.

Once packed, I dropped the server off at a nearby shipping depot. It arrived in the datacenter a few days later, undamaged (yay!), and a few days after that, the datacenter had racked it and plugged it in.

Distributed systems have a well-known failure mode, referred to as split-brain. This is when two parts of the network stop communicating, and their states evolve differently over time. Once the nodes reconnect, they don't know how to reconcile their state differences, and bad things can happen.

Proxmox is pretty robust against this. It has a voting system such that, if the network splits into pieces, a maximum of only one section maintains a quorum. The other sections will become severely limited, blocking most VM actions. This includes starting and stopping VMs, migrations, and most everything else.

Although this is generally very sensible, in our case we need to purposefully disable these checks and balances. VMs need to be started on Node A in NJ, to allow for routing and tunnelling.

I got around this by modifying each node individually to tell it that it controlled the majority of the votes on its own. This allowed both nodes to think they had a quorum. This is definitely a terrible idea but I did a bit of testing to ensure nothing would break before actually doing it to my cluster. I strongly recommend avoiding doing this if you don't know what you're doing.

Now that I had two functioning nodes, I had to tunnel them together. The two unrelated networks need to act like they're bridged, so that as I migrate VMs, everything can still communicate. I decided to use OpenVPN for the actual tunnel, combined with ARP proxying to allow the VMs to talk to each other.

ARP, or Address Resolution Protocol, is basically how network devices convert an IP address to a MAC address, which can then be used for communication. At a high level, a machine might call out for a specific IP, and the machine holding that IP would respond with its MAC address.

The trick with ARP proxying is to have the two nodes of our VPN respond to ARP requests destined for machines on the other end of the tunnel. That way, packets will end up going to the relevant VPN node, forwarded across the tunnel, and re-broadcasted on the other end. This allows the network to appear bridged, but it also means we need to update the ARP configuration after every migration. Also, "real networking people" (tm) told me this was a bad idea, and they're right. The better solution would be to have two different subnets on either of the two networks. This complicates migration, though - instead of just updating an ARP table, I'd need to update the VM's IP address.

Since the VPN nodes run NixOS, I was able to set up my config such that I had two arrays representing the IPs on either end of the link. After every migration, I would just move the relevant IP between the arrays, and re-deploy the two VPN nodes. It was pretty straightforward, but it also meant I had to be available when the migration finished to run the redeployment. This was a little hard to coordinate and did lead to at least one subway migration. (Side note - I think "subway migration" would be a good square on a DevOps Bingo card).

I slowly migrated everything over the Internet over the span of about a week. This also involved a few DNS changes to point my domains at the NJ IP address instead of my home IP. The last step was destroying the cluster and shutting down Node B, still at my Canadian residence. In the next few weeks I'll be returning to get that, and probably redeploy it in my apartment in NYC.

Things went well! I didn't have more than a few minutes of downtime per VM. I was pretty nervous about something going catastrophically wrong and causing everything to break, but fortunately that didn't happen. I also learned a ton about networking, especially in relation to OpenVPN (which I probably hadn't touched in about 10 years). I'm excited to finally have my gear in a reliable colo, where I hopefully can be a lot less worried about something going wrong.

At the same time, there are certainly some things I still need to change. The tunnel is still active; I use it to bridge my GPU server and backup server to the datacenter. I really need to do things properly and use two separate subnets instead of an ARP proxy. I'm hoping to get to that sometime after I move the rest of my hardware to NYC.