NixOS root on zfs: /dev/sda1 already mounted

[onsails]

I've precisely followed instructions here. However nixos-install --root /mnt --no-root-passwd -v --show-trace fails with the following error.

/nix/store/bwy7h76zw6hl3b7n45fwaa9cf2dh0v9n-nixos-system-nixos-22.05.2361.45c9736ed69
installing the boot loader...
setting up /etc...
/etc/tmpfiles.d/journal-nocow.conf:26: Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
updating GRUB 2 menu...
mount: /boot/efis/ata-nixos-aarch64-0_SSD_HPTWB1HVZVAM31NANS7Q-part1: /dev/sda1 already mounted on /boot/efis/ata-nixos-aarch64-0_SSD_HPTWB1HVZVAM31NANS7Q-part1.
Died at /nix/store/ids166d0qf8k4s7clmwzbba2ml4ysxik-install-grub.pl line 608.

Indeed, as a result of step 5 /dev/sda gets mounted:

[root@nixos:~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        149M     0  149M   0% /dev
tmpfs           1.5G     0  1.5G   0% /dev/shm
tmpfs           743M  9.4M  734M   2% /run
tmpfs           1.5G  432K  1.5G   1% /run/wrappers
tmpfs           1.5G   26M  1.5G   2% /
/dev/root       802M  802M     0 100% /iso
tmpfs           1.5G  147M  1.4G  10% /nix/.rw-store
overlay         1.5G  147M  1.4G  10% /nix/store
tmpfs           298M   12K  298M   1% /run/user/1000
/dev/sda1      1022M  4.0K 1022M   1% /mnt/boot/efi

[ghost]

Thanks for reporting the issue. However, it seems that the ``already mounted'' message is harmless and the real culprit is elsewhere, here's the output on my computer:

***@***.***:~]# nixos-rebuild boot --install-bootloader
building Nix...
building the system configuration...
updating GRUB 2 menu...
mount: /boot/efis/ata-MK0100GCTYU_BTTV508202HV100FGN-part1: /dev/sda1 already mounted on /boot/efis/ata-MK0100GCTYU_BTTV508202HV100FGN-part1.
installing the GRUB 2 boot loader on /dev/disk/by-id/ata-MK0100GCTYU_BTTV508202HV100FGN...
Installing for i386-pc platform.
Installation finished. No error reported.
installing the GRUB 2 EFI boot loader into /boot/efi...
Installing for x86_64-efi platform.
Installation finished. No error reported.

On my x86-64 computer the line 608 of install-grub.pl is related to EFI support. What about yours? Your computer appears to be an aarch64 one.

Andrey Kuznetsov @.***> writes:

I've precisely followed instructions here. However nixos-install --root /mnt --no-root-passwd -v --show-trace fails with the following error.

/nix/store/bwy7h76zw6hl3b7n45fwaa9cf2dh0v9n-nixos-system-nixos-22.05.2361.45c9736ed69
installing the boot loader...
setting up /etc...
/etc/tmpfiles.d/journal-nocow.conf:26: Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
updating GRUB 2 menu...
mount: /boot/efis/ata-nixos-aarch64-0_SSD_HPTWB1HVZVAM31NANS7Q-part1: /dev/sda1 already mounted on /boot/efis/ata-nixos-aarch64-0_SSD_HPTWB1HVZVAM31NANS7Q-part1.
Died at /nix/store/ids166d0qf8k4s7clmwzbba2ml4ysxik-install-grub.pl line 608.

Indeed, as a result of step 5 /dev/sda gets mounted:

***@***.***:~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        149M     0  149M   0% /dev
tmpfs           1.5G     0  1.5G   0% /dev/shm
tmpfs           743M  9.4M  734M   2% /run
tmpfs           1.5G  432K  1.5G   1% /run/wrappers
tmpfs           1.5G   26M  1.5G   2% /
/dev/root       802M  802M     0 100% /iso
tmpfs           1.5G  147M  1.4G  10% /nix/.rw-store
overlay         1.5G  147M  1.4G  10% /nix/store
tmpfs           298M   12K  298M   1% /run/user/1000
/dev/sda1      1022M  4.0K 1022M   1% /mnt/boot/efi

-- Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs-docs/issues/337 You are receiving this because you are subscribed to this thread.

Message ID: @.***>

[onsails]

Hmm got it.

My instalation dies on this line:

        if (($grubTarget eq "") || ($grubTargetEfi eq "")) { die }

Will investigate.

[ghost]

I think that the bootloader, GRUB, is currently not compatible with your ARM computer and thus can not be used at all.

I have heard that the bootloader situation on ARM is rather difficult where each board requires its own board-specific disk image to boot. GRUB does not support this.

There are two options, IMO:

1. Ditch GRUB and bpool, and roll your own Root on ZFS guide.

2. Try to install UEFI, which abstracts away the board specific details, then follow the original guide here, with GRUB.

   See https://nixos.wiki/wiki/NixOS_on_ARM/UEFI for details.

[ghost]

Tow-boot project seems to be able to solve the problem at hand:

Check if your device is supported:

https://tow-boot.org/devices/index.html

If supported, install it on your device:

https://tow-boot.org/getting-started.html

Download unstable or 22.05 release ARM64 disk image:

unstable: https://hydra.nixos.org/job/nixos/trunk-combined/nixos.iso_minimal.aarch64-linux

22.05 release, older packages: https://hydra.nixos.org/job/nixos/release-22.05-aarch64/nixos.iso_minimal.aarch64-linux

Then follow the guide as-is, with the exception of disabling BIOS boot with:

boot.loader.grub.device = "nodev";

You might need to install the Root on ZFS system to a disk other than the UEFI boot disk.

[onsails]

Thank you for your suggestions! I'm installing on Parallels and apparently tow boot doesn't support it. Will try using uefi.

[onsails]

Apparently Parallels is already using uefi by default:

[nixos@nixos:~]$ ls /sys/firmware/efi/
efivars  fw_platform_size  systab

[ghost]

You might need to check that the disk image used is indeed the generic, UEFI-enabled one. According to the wiki, GRUB UEFI for ARM should Just Work under these circumstances.

However, I think running Root on ZFS with your setup comes with few benefits but some notable drawbacks.

• AFAIK, Apple's APFS is a Copy-on-Write filesystem.

• I do not know how Parallels internally manages virtual disks but it likely supports taking snapshots for VMs, which is indicative of support for CoW.

• ZFS is also CoW. Performance of CoW-on-CoW is not good.

• ZFS features like resilency against corruption, disk array, redundancy, etc. will not be of much use for you since the underlying storage is only one single disk.

• The snapshot feature is provided by the hypervisor, i.e. Parallels.

So I think Root on ZFS might be doable, but that does not mean that one should do it on such a setup.

[onsails]

I don't care of ZFS on APFS drawbacks because actually what I'm trying to do is – install NixOS on a USB SSD from Parallels, then plug this SSD into raspberry pi and make it boot from this SSD.

I was using 22.05-aarch (https://hydra.nixos.org/job/nixos/release-22.05-aarch64/nixos.iso_minimal.aarch64-linux) image which is UEFI-compatible so I don't have an idea why it fails...

NixOS manual says that:

UEFI systems You must set the option boot.loader.systemd-boot.enable to true. nixos-generate-config should do this automatically for new configurations when booted in UEFI mode.

However nixos-generate-config doesn't do this automatically. Hmm.

[ghost]

Andrey Kuznetsov @.***> writes:

I don't care of ZFS on APFS drawbacks because actually what I'm trying to do is – install NixOS on a USB SSD from Parallels, then plug this SSD into raspberry pi and make it boot from this SSD.

Well I see that Tow Boot does support raspi. It might work if we skip the Parallels step.

I was using 22.05-aarch (https://hydra.nixos.org/job/nixos/release-22.05-aarch64/nixos.iso_minimal.aarch64-linux) image which is UEFI-compatible so I don't have an idea why it fails...

NixOS manual says that:

UEFI systems You must set the option boot.loader.systemd-boot.enable to true. nixos-generate-config should do this automatically for new configurations when booted in UEFI mode.

However nixos-generate-config doesn't do this automatically. Hmm.

It should be mentioned that all non-GRUB bootloader options auto-generated are removed in this Root on ZFS guide, as they conflict with GRUB.

What happens if you add the option back manually, delete GRUB related options then install NixOS?

-- Reply to this email directly or view it on GitHub: https://github.com/openzfs/openzfs-docs/issues/337#issuecomment-1219836602 You are receiving this because you commented.

Message ID: @.***>

[onsails]

What happens if you add the option back manually, delete GRUB related options then install NixOS?

nixos-install completes successfully!

Now the challenge is to boot raspberry-pi itself.

1. I've enabled usb-boot (via program_usb_boot_mode=1 in config.txt, I'm using model 3 b+).
2. Plug the SSD with nixos installed
3. Power raspberry.

Get "Failed to start Remount root and Kernel File Systems".

I suppose I have to install tow-boot on the front of an SSD, then install nixos on the rest of the SSD?

[onsails]

Hmm. What's weird that if I burn SSD with an official raspberry os image, it gets booted, resizes partition (like it usually does on first boot), then reboots and also fails with "Failed to start Remount root and Kernel File Systems".

[ghost]

systemd-boot was not used in the guide, because it copies kernels into the FAT32-formatted EFI system partition and thus unprotect against file corruption.

But in your case, you are not using any disk array for redundancy anyway, so that point is moot here.

The next problem, I guess, is to get tow-boot installed and booted, on another storage device like an SD card or internal emmc, per the Getting Started guide.

A separate storage device is needed because tow-boot comes with its own partition table, which is incompatible with the ZFS partition table.

After that, you can theoretically just plug in and select the ZFS disk as boot device as you would normally do on any x86 PC.

[ghost]

Hello. The guide has been updated to support x86_64 based computers exclusively. See here

https://github.com/openzfs/openzfs-docs/blob/cc6d72c02db6f36136be3f4b7ae273b8271333a7/docs/Getting%20Started/NixOS/Root%20on%20ZFS/configuration.nix#L199

ARM support is now out-of-scope.

[gmelikov]

Looks like not actual, boot process for ARMs is a different hard topic which is not supported in our docs for now, closed. Feel free to open new issues or contribute things on such topic.

[ghost]

I got an OrangePi PC2, which is supported by Tow-Boot, and was able to test ARM installation of NixOS Root on ZFS. Some observations:

• Tow-Boot must be installed first on either SD card, internal eMMC or SPI flash. As some devices, including mine, do not have internal eMMC or SPI flash, SD card is the only option.
• Then, for best performance, it is recommended to leave the SD card alone, using it only and solely for Tow-Boot firmware (5MB in size), and install NixOS somewhere else, such as an external solid-state drive, connected with an USB enclosure. This way you get much better reading and writing performance than any SD card would provide, and we can avoid accidentally modifying the SD card partition table in any way.
• Now, besides an SD card, an external USB enclosure with the installation target disk, you will also need another USB drive containing NixOS live image itself. Just download latest stable live UEFI-ISO image for the Aarch 64 architecture on the same page as x86_64 and write it to the USB drive.
• Insert SD card with Tow-Boot to your single-board computer, plug in the two external USB devices. Then power on. At Tow-Boot boot menu, select the device containing the live NixOS image.
• Finally, follow the updated guide verbatim. Installation procedures for x86_64 and Aarch64 on UEFI is identical.
• Note: On the last page of the guide, before applying configuration, you are advised to change system type from x86_64-linux to aarch64-linux inside the template configuration.

Now you should have a fully functional NixOS ARM64 Root on ZFS system at your disposal.

It is possible to install NixOS on the same SD card as Tow-Boot, but it involves some hairy partition table modifications, prevents easy upgrade of Tow-Boot firmware, and complicates the Root on ZFS installation by not following the partition order in the guide. Performance is also poor.

[ghost]

Update: I've fixed this error

        if (($grubTarget eq "") || ($grubTargetEfi eq "")) { die }

in the new template configuration repo for ARM64-UEFI devices.

https://github.com/ne9z/dotfiles-flake/blob/2c6ec144f2c71cbd4865f11acd68333eeb2b06df/modules/boot/grub-for-arm64/install-grub.pl#L610