A Btrfs Change in Linux 7.2 Quietly Broke VPN Logins, and 7.3-rc4 Reverts It

A Btrfs Change in Linux 7.2 Quietly Broke VPN Logins, and 7.3-rc4 Reverts It

Two Btrfs fixes merged on September 19 ahead of Linux 7.3-rc4, and the first is a good demonstration of how an internal-looking change becomes a user-facing outage.

The f_fsid regression

In Linux 7.2-rc1, Btrfs changed how it derives f_fsid, the filesystem identifier returned by statfs().

The consequence was that the value stopped being stable. It could change after a kernel update, and it could shift across reboots. Anything that had treated f_fsid as a durable identifier for a filesystem now saw a different number for the same disk.

The visible casualty was VPN access. Some applications use the FSID as an input when encrypting and decrypting stored key material, on the reasonable assumption that a filesystem identifier identifies a filesystem. When the identifier changed, the key files no longer decrypted. NetworkManager OpenConnect users on Btrfs who upgraded to 7.2 lost the ability to connect, with nothing obviously pointing at the filesystem as the cause.

7.3-rc4 reverts to deriving the FSID from the filesystem UUID, which is stable by construction.

The uncomfortable part is the gap. This landed in 7.2-rc1 and is only being fixed now, which means it shipped in stable 7.2 and reached users. Anyone still on 7.2 with a Btrfs root and an affected VPN setup is living with it until the fix reaches their kernel.

There is a general lesson here that applies well beyond Btrfs. If a value is exposed to userspace, it is an API, whether or not it was intended as one. f_fsid is not documented as stable across boots, and the Linux kernel’s actual rule is not what the documentation says. It is that you do not break working userspace. The revert is the correct outcome under that rule.

The GRUB2 boot fix

The second bug was introduced during the 7.3 cycle itself, so it never reached a stable release.

Btrfs broke detection of /dev/root, and the specific victim was booting without an initramfs. GRUB2 with a Btrfs root filesystem and no initramfs could not identify the root device.

Initramfs-less boot is a minority configuration and a deliberate one. People choose it to cut boot time and to remove a moving part, since the initramfs is an entire second filesystem that has to be regenerated whenever the kernel or its modules change. Our initramfs guide covers what it actually does and why skipping it is viable when your root filesystem driver is built into the kernel rather than loaded as a module.

Being caught inside the rc cycle is the system working. The f_fsid one is the system not working.

Both fixes

# the merge
git show 518e5b794c06c0f0eb40df3e202274a66202c137

# check your own fsid stability across reboots
stat -f --format="%i" /

If that value changes after a reboot on 7.2, you have the regression.

Who should care

Btrfs users on 7.2 with anything keying off filesystem identity: VPN clients storing encrypted credentials, licence managers, backup tools that fingerprint a filesystem rather than reading its UUID directly.

Anyone booting Btrfs without an initramfs should skip 7.3 release candidates before rc4.

Neither bug involves data loss, which is worth saying plainly given how much of this cycle has been filesystem churn. Our Btrfs versus ZFS comparison covers the wider tradeoffs, and Btrfs snapshots with Timeshift covers the rollback path if a kernel update does leave you somewhere you do not want to be.

Background reading

Explainers for the concepts behind this story.