phantom-node
Native ZFS encryption support (Cryptrebooting fails with "No such file or directory @ dir_chdir")
Trying to cryptreboot fails with the following error:
Extracting initramfs... To speed things up, future versions will employ cache.
Running (echo lz4 \"-t\"; strace -f --trace=execve -z -qq --signal=\!all unmkinitramfs /boot/initrd.img-6.8.12-10-pve /tmp/d20250503-1661553-u8qwpo) 2>&1 | grep --line-buffered lz4
lz4 "-t"
Finished in 12.978 seconds with exit status 0 (successful)
Running mount -t ramfs -o mode\=700 none /tmp/d20250503-1661553-xgpdqm
Finished in 0.005 seconds with exit status 0 (successful)
Running umount /tmp/d20250503-1661553-xgpdqm
Finished in 0.004 seconds with exit status 0 (successful)
/var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/initramfs/archiver.rb:10:in `chdir': No such file or directory @ dir_chdir - /tmp/d20250503-1661553-xgpdqm/files (Errno::ENOENT)
from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/initramfs/archiver.rb:10:in `call'
from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/initramfs/patcher.rb:14:in `block in call'
from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/safe_temp/directory.rb:13:in `block (2 levels) in call'
from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/safe_temp/mounter.rb:17:in `run'
from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/safe_temp/mounter.rb:11:in `call'
from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/safe_temp/directory.rb:12:in `block in call'
from /usr/lib/ruby/3.1.0/tmpdir.rb:96:in `mktmpdir'
from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/safe_temp/directory.rb:11:in `call'
from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/safe_temp/directory.rb:11:in `call'
from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/initramfs/patcher.rb:11:in `call'
from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/patched_initramfs_generator.rb:8:in `call'
from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/kexec_patching_loader.rb:12:in `call'
from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/cli/params_parsing_executor.rb:25:in `configure_and_exec'
from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/cli/params_parsing_executor.rb:9:in `call'
from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/exe/cryptreboot:8:in `<top (required)>'
from /usr/local/bin/cryptreboot:25:in `load'
from /usr/local/bin/cryptreboot:25:in `<main>'
tmp is mounted
rpool/var/tmp on /var/tmp type zfs (rw,relatime,xattr,posixacl,casesensitive)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=65625572k,nr_inodes=409600,inode64)
various kernel (pve / proxmox) are present:
config-5.13.19-6-pve efi initrd.img-6.8.12-10-pve System.map-5.13.19-6-pve vmlinuz-5.13.19-6-pve
config-5.15.30-1-pve efi2 initrd.img-6.8.12-4-pve System.map-5.15.30-1-pve vmlinuz-5.15.30-1-pve
config-6.8.12-10-pve grub initrd.img-6.8.12-9-pve System.map-6.8.12-10-pve vmlinuz-6.8.12-10-pve
config-6.8.12-4-pve initrd.img-5.13.19-6-pve lost+found System.map-6.8.12-4-pve vmlinuz-6.8.12-4-pve
config-6.8.12-9-pve initrd.img-5.15.30-1-pve pve System.map-6.8.12-9-pve vmlinuz-6.8.12-9-pve
System is a fully updated bookworm / proxmox.
Let me know if should provide further information to debug the issue.
You're not asked for a passphrase. You should be at this point. Probably cryptreboot couldn't understand how your drive is encrypted. Please provide the output of the following commands:
cat /etc/crypttab
ls -l /dev/mapper
cat /etc/fstab
mount
OK I, realized that I'm probably missing an important bit of information. I could imagine that detection of the actual volume to unlock fails due to my non-standard setup.
The system has two ZFS pools. One backed by SSDs which use ZFS native encryption. This pool is used for booting and the system. There's a second pool backed by spinning disks (luks encrypted for historic reasosn) used for data. The second pool backing disks are unlocked later in the boot process using keyfiles.
# ls /dev/zvol/
datapool rpool
Crypttab to unlock HDDs using a key stored on the encrypted zfs volume on SSDs.
# cat /etc/crypttab
hdd3zfscrypt /dev/disk/by-id/ata-WDC_WD80EFAX-68KNBN0_VAHEBWWL /etc/hddkeyfile luks
hdd4zfscrypt /dev/disk/by-id/ata-WDC_WD80EFAX-68KNBN0_VAHEESAL /etc/hddkeyfile luks
Mapping of these volumes
# ls -l /dev/mapper
total 0
crw------- 1 root root 10, 236 May 3 14:24 control
lrwxrwxrwx 1 root root 7 May 3 14:24 hdd3zfscrypt -> ../dm-0
lrwxrwxrwx 1 root root 7 May 3 14:24 hdd4zfscrypt -> ../dm-1
fstab has efi and boot and swap partitions only...
total 0
crw------- 1 root root 10, 236 May 3 14:24 control
lrwxrwxrwx 1 root root 7 May 3 14:24 hdd3zfscrypt -> ../dm-0
lrwxrwxrwx 1 root root 7 May 3 14:24 hdd4zfscrypt -> ../dm-1`
mount output
# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=65591468k,nr_inodes=16397867,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=13125720k,mode=755,inode64)
rpool/ROOT/debian on / type zfs (rw,relatime,xattr,posixacl,casesensitive)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5403)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
rpool/home on /home type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/home/root on /root type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/proxmox on /rpool/proxmox type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/var/cache on /var/cache type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/var/log on /var/log type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/var/nfs on /var/nfs type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/var/spool on /var/spool type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/var/tmp on /var/tmp type zfs (rw,relatime,xattr,posixacl,casesensitive)
/dev/sda3 on /boot type ext4 (rw,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=65628592k,nr_inodes=409600,inode64)
/dev/sda2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/sdb2 on /boot/efi2 type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
datapool/proxmox on /datapool/proxmox type zfs (rw,noatime,xattr,noacl,casesensitive)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=13125716k,nr_inodes=3281429,mode=700,inode64)
Thanks. I still don't see /etc/fstab contents, it seems you duplicated ls -l /dev/mapper output there.
Using ZFS is an important piece of information. Could you provide me with the output of:
zfs list
On Ubuntu 24.04 it looks like this:
NAME USED AVAIL REFER MOUNTPOINT
bpool 97.3M 1.40G 96K /boot
bpool/BOOT 96.8M 1.40G 96K none
bpool/BOOT/ubuntu_awvp0x 96.7M 1.40G 96.7M /boot
rpool 3.97G 6.68G 192K /
rpool/ROOT 3.94G 6.68G 192K none
rpool/ROOT/ubuntu_awvp0x 3.94G 6.68G 2.86G /
rpool/ROOT/ubuntu_awvp0x/srv 192K 6.68G 192K /srv
rpool/ROOT/ubuntu_awvp0x/usr 580K 6.68G 192K /usr
rpool/ROOT/ubuntu_awvp0x/usr/local 388K 6.68G 388K /usr/local
rpool/ROOT/ubuntu_awvp0x/var 1.08G 6.68G 192K /var
rpool/ROOT/ubuntu_awvp0x/var/games 192K 6.68G 192K /var/games
rpool/ROOT/ubuntu_awvp0x/var/lib 1.07G 6.68G 976M /var/lib
rpool/ROOT/ubuntu_awvp0x/var/lib/AccountsService 212K 6.68G 212K /var/lib/AccountsService
rpool/ROOT/ubuntu_awvp0x/var/lib/NetworkManager 224K 6.68G 224K /var/lib/NetworkManager
rpool/ROOT/ubuntu_awvp0x/var/lib/apt 75.4M 6.68G 75.4M /var/lib/apt
rpool/ROOT/ubuntu_awvp0x/var/lib/dpkg 47.8M 6.68G 47.8M /var/lib/dpkg
rpool/ROOT/ubuntu_awvp0x/var/log 4.16M 6.68G 4.16M /var/log
rpool/ROOT/ubuntu_awvp0x/var/mail 192K 6.68G 192K /var/mail
rpool/ROOT/ubuntu_awvp0x/var/snap 2.16M 6.68G 2.16M /var/snap
rpool/ROOT/ubuntu_awvp0x/var/spool 244K 6.68G 244K /var/spool
rpool/ROOT/ubuntu_awvp0x/var/www 192K 6.68G 192K /var/www
rpool/USERDATA 6.10M 6.68G 192K none
rpool/USERDATA/home_r9xgh7 5.46M 6.68G 5.46M /home
rpool/USERDATA/root_r9xgh7 464K 6.68G 464K /root
rpool/keystore 22.5M 6.69G 16.5M -
There is a keystore volume at the bottom. It contains an LUKS-encrypted ext4 filesystem. Do you have an equivalent of that?
I don't see it in your /dev/mapper directory and mount output, but maybe it's just not unlocked and not currently mounted.
Sure thing. I fixed the output in my previous comment to show the command line. /dev/mapper and mount was already included.
fstab
# cat /etc/fstab
/dev/disk/by-uuid/21e4d148-156f-4130-9da2-d3fc29891f27 /boot ext4 defaults 0 0
/dev/disk/by-uuid/51D4-EF56 /boot/efi vfat defaults 0 0
/dev/disk/by-uuid/8E8E-DAC5 /boot/efi2 vfat defaults 0 0
/dev/zvol/rpool/swap none swap defaults 0 0
ZFS list
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
datapool 4.30T 2.84T 96K /data
datapool/home 4.07T 2.84T 96K /data/home
datapool/proxmox 232G 2.84T 96K /datapool/proxmox
datapool/proxmox/vm-102-disk-1 232G 2.94T 82.0G -
rpool 311G 139G 192K none
rpool/ROOT 143G 139G 160K none
rpool/ROOT/debian 143G 139G 31.1G /
rpool/home 57.3G 139G 360K /home
rpool/proxmox 90.5G 139G 200K /rpool/proxmox
rpool/proxmox/vm-101-disk-0 90.5G 171G 7.53G -
rpool/swap 16.3G 139G 16.3G -
rpool/var 3.20G 139G 296K none
rpool/var/cache 125M 139G 76.1M /var/cache
rpool/var/log 3.06G 139G 849M /var/log
rpool/var/nfs 176K 139G 176K /var/nfs
rpool/var/spool 16.3M 139G 3.05M /var/spool
rpool/var/tmp 536K 139G 292K /var/tmp
I don't have a keystore volume or equivalent.
Thanks. Cryptreboot currently supports LUKS encryption only, the ZFS support uses LUKS underneath.
On Ubuntu, when ZFS is chosen during installation, creates a special keystore volume. This volume contains key material for the native ZFS encryption and is encrypted with LUKS. On boot, this keystore volume is decrypted. This allows retrieval of a key to unlock the ZFS root filesystem.
As your system doesn't contain the keystore or equivalent, the current version of cryptreboot won't be able to unlock it.
Unfortunately, I don't think I will be able to implement this feature in the near future due to a lack of time :/
I don't have any experience with Proxmox, but I did some preliminary research. I installed Proxmox VE 8.4, however, it seems it doesn't support activating disk encryption during the installation. On this page I found following sentence:
There is currently no support for booting from pools with encrypted datasets using GRUB, and only limited support for automatically unlocking encrypted datasets on boot.
Did you use a guide or tutorial to set up the ZFS system encryption? It would be helpful if you shared it. This way, I could try to implement this feature when I have some free time.
Thanks Pawel, I really appreciate you taking time digging into this and for the explanation. Don't hesitate to deprioritize this feature for now.
I essentially have a set up with limited support for automatically unlocking encrypted datasets on boot.. My boot partition is unencrypted . I had an encrypted boot partition on a separate encrypted ZFS pool before but that was a pain to maintain (grub is incompatible with certain ZFS pool features).
Regardingn installation, I recall following the ZFS docs for debian in the past but the system has been upgraded multiple times.
That's no problem at all, and thanks for the understanding and the context! I appreciate you sharing the details about your setup and the link to the ZFS docs for Debian—that's helpful background.
Did dig a bit deeper into the rabbithole and it seems as there is no standard mechanism yet to provide the passphrase for key decryption in initramfs. https://github.com/openzfs/zfs/issues/13757 has some context.
I checked my /usr/share/initramfs-tools in particular the contained zfsunlock script but didn't see a way to inject the passphrase.
My installed zfs-initramfs version is
zfs-initramfs/stable,now 2.2.7-pve2 all [installed]
OpenZFS root filesystem capabilities for Linux - initramfs
Thanks for the interesting info. The issue you linked explains the Ubuntu approach.
My current idea is to rewrite cryptreboot to replace zfs binary with a script calling the renamed original file. The script would inject the key material.
Of course, the zfs would be replaced only in the in-memory patched initramfs.
I believe this approach would free cryptreboot from having to know the (possibly changing) internals of the ZFS unlocking implementation in each distro.
What might even be easier than replacing the binary is to prepare an initramfs hook script that unlocks the pools. One option could be to to read out pools with loaded keys as a first step. Then ask for the passphrases which are then used to forge the hook script. The zfs unlock process as far as I can see is only used if the pool is not unlocked.
Do you have an existing testbed for distros / test cases?
Actually, not hook scripts but boot scripts would help. Hook scripts interact with initramfs building as far as I understood. So you might be right in that it's easier to patch the zfs script and wrap into a loader for keys.