envfs with initrd.systemd: "unpopulated /usr/" error

[richardgill]

My set up uses services.envfs.enable = true + btrfs impermanence together.

I have a script which uses btrfs to wipe the root volume when the machine boots:

https://github.com/richardgill/nix/blob/main/modules/system/nixos/headless/impermanence.nix#L17-L37

# Wipe script that rolls back btrfs root subvolume
mkdir /btrfs_tmp
mount -t btrfs -o subvol=/ /dev/mapper/cryptroot /btrfs_tmp
btrfs subvolume list -o /btrfs_tmp/root | cut -f9 -d' ' | \
while read subvolume; do
  btrfs subvolume delete "/btrfs_tmp/$subvolume"
done
btrfs subvolume delete /btrfs_tmp/root
# this wipes everything in the root subvolume
btrfs subvolume snapshot /btrfs_tmp/root-blank /btrfs_tmp/root
umount /btrfs_tmp

The script is executed on the boot.initrd.postResumeCommands. This works great with envfs.

I needed to switch to boot.initrd.systemd.enable = true to enable secure boot (specifically for use with lanzaboote).

I do this by running boot.initrd.systemd.services.rollback https://github.com/richardgill/nix/blob/main/modules/system/nixos/headless/impermanence.nix#L138-L146

boot.initrd.systemd.services.rollback = {
  description = "Rollback btrfs root subvolume to pristine state";
  wantedBy = [ "initrd.target" ];
  after = [ "[email protected]" ];
  before = [ "sysroot.mount" ];
  unitConfig.DefaultDependencies = "no";
  serviceConfig.Type = "oneshot";
  script = <script above>
}

Running the same script with systemd results in the boot failing.

Error from boot:

[!!!!!!] Refusing to run in unsupported environment where /usr/ is not populated.
4.862770] systemd[1]: Freezing execution.

The system hangs at "Starting Switch Root..." and drops to this error, then freezes.

I found a post here with a ~related issue: https://discourse.nixos.org/t/impermanence-25-05-not-populating-usr/64567/7

I discovered that setting services.envfs.enable = false fixes the issue.

Workaround

I found a workaround by @linyinfeng: https://github.com/linyinfeng/dotfiles/blob/main/nixos/profiles/services/envfs/default.nix

https://github.com/richardgill/nix/blob/main/modules/system/nixos/headless/compat.nix

services.envfs.enable = true;

# Workaround only applies when systemd in initrd is enabled

boot.initrd.systemd.tmpfiles.settings = lib.mkIf config.boot.initrd.systemd.enable {
  "50-usr-bin" = {
    "/sysroot/usr/bin" = {
      d = {
        group = "root";
        mode = "0755";
        user = "root";
      };
    };
  };
};

# linyinfeng also has these, but in my case they're not needed.
fileSystems."/usr/bin".options = lib.mkIf config.boot.initrd.systemd.enable [
  "[email protected]"
  "[email protected]"
];
fileSystems."/bin".enable = lib.mkIf config.boot.initrd.systemd.enable false;

I don't fully understand how @linyinfeng's workaround works! I'm not sure if this is a bug in envfs, or if there's a better way I could write my rollback script (or perhaps time it better?)

Any help would be greatly appreciated. Thanks, Richard.