Incus 7.4 Adds Near-Live Container Migration for ZFS and Btrfs

Incus 7.4 Adds Near-Live Container Migration for ZFS and Btrfs

Incus 7.4 adds near-live container migration for ZFS and Btrfs storage backends.

What near-live means here

Migrating a running container between hosts requires moving its filesystem and its process state without the container noticing more than it can tolerate.

The filesystem is the slow part. Copying tens of gigabytes while the container keeps writing means the copy is stale before it finishes.

ZFS and Btrfs both solve this with incremental send and receive. Snapshot the dataset, send it to the target while the container keeps running, then snapshot again and send only the difference, which is small because it covers seconds rather than the whole history. Repeat until the increment is tiny, then pause the container, send the final delta, and resume on the target.

# the underlying mechanism, roughly
zfs send pool/container@snap1 | ssh target zfs receive pool/container
zfs send -i @snap1 pool/container@snap2 | ssh target zfs receive pool/container

The pause is the time to transfer the last increment plus restoring process state, which is short enough that most workloads survive it without dropping connections.

Our ZFS and Btrfs snapshots explainers cover the underlying filesystem features, and this is a good example of why copy-on-write filesystems are worth the complexity: the snapshot mechanism you installed for backups turns out to enable migration too.

What Incus is

Incus is the community fork of LXD, created after Canonical moved LXD under a CLA and relicensed it. It is maintained by Stéphane Graber, LXD’s original lead, and is now in Debian and most other distributions.

It manages system containers and virtual machines, which is a different thing from Docker.

A Docker container runs one process and is designed to be disposable. An Incus system container runs a full init system with multiple services and behaves like a lightweight VM you can SSH into, with the isolation of a container and none of the hypervisor overhead.

For running several isolated environments on one machine, that model is frequently a better fit than either Docker or full virtualisation. Our LXC and Incus guide covers the distinction, and Docker and Podman basics covers the application container model for comparison.

Why migration matters at this scale

Live migration is normally enterprise virtualisation territory: vMotion, Proxmox clusters, oVirt. Having it work well for containers on a self-hosted setup with two machines is a different proposition.

The practical use is host maintenance. Move the containers off, reboot the host for a kernel update, move them back. Without migration that is planned downtime; with it, it is an operation nobody notices.

For a homelab running services people in the house depend on, that is the difference between patching promptly and putting it off.

Proxmox VE remains the more complete option if you want a management interface and cluster features out of the box. Incus is the better fit if you want containers as the primary unit and are comfortable on the command line.

Background reading

Explainers for the concepts behind this story.