DietPi icon indicating copy to clipboard operation
DietPi copied to clipboard

ZFS problem

Open Seb-tech77 opened this issue 1 year ago • 15 comments

Hello, I use openmediavault to manage my disks connected to my orange pi 5 pro. I wanted to switch my disk to zfs to benefit from data deduplication but when I want to use the OMV extension I get this error:

OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin ; export LC_ALL=C.UTF-8 ; export LANGUAGE= ; zfs list -p -H -t all -o name,type 2>&1' with exit code '1': ZFS modules are not loaded.
Try running '/sbin/modprobe zfs' as root to load them. in /usr/share/php/openmediavault/system/process.inc:247
Stack trace :
#0 /usr/share/omvzfs/Utils.php(450) : OMV\System\Process->execute()
#1 /usr/share/omvzfs/Utils.php(262) : OMVModuleZFSUtil::exec()
#2 /usr/share/openmediavault/engined/rpc/zfs.inc(297) : OMVModuleZFSUtil::getZFSFlatArray()
#3 [internal function]: OMVRpcServiceZFS->listPools()
#4 /usr/share/php/openmediavault/rpc/serviceabstract.inc(124) : call_user_func_array()
#5 /usr/share/php/openmediavault/rpc/serviceabstract.inc(155) : OMV\Rpc\ServiceAbstract->callMethod()
#6 /usr/share/php/openmediavault/rpc/serviceabstract.inc(628) : OMV\Rpc\ServiceAbstract->OMV\Rpc\{closure}()
#7 /usr/share/php/openmediavault/rpc/serviceabstract.inc(152) : OMV\Rpc\ServiceAbstract->execBgProc()
#8 /usr/share/openmediavault/engined/rpc/zfs.inc(304) : OMV\Rpc\ServiceAbstract->callMethodBg()
#9 [internal function] : OMVRpcServiceZFS->listPoolsBg()
#10 /usr/share/php/openmediavault/rpc/serviceabstract.inc(124): call_user_func_array()
#11 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod()
#12 /usr/sbin/omv-engined(544): OMV\Rpc\Rpc::call()
#13 {main}

Could you tell me if this error comes directly from openmediavault or I also saw that it could be an incompatibility between the linux kernel and the ZFS module?

Sincerely

Seb-tech77 avatar Jan 22 '25 14:01 Seb-tech77

Did you install/compile the ZFS kernel module? This is not part of the kernel by default, but needs to be added via DKMS, which also requires kernel headers. Which kernel do you have installed?

dpkg -l | grep linux-image

MichaIng avatar Jan 22 '25 15:01 MichaIng

No, I haven't tried to compile the module for the kernel, I don't really know how to do it.

Sebastien@DietPi:~$ dpkg -l | grep linux-image ii linux-image-vendor-rk35xx 25.02.0-trunk-dietpi1 arm64 Armbian Linux vendor kernel image 6.1.84-vendor-rk35xx Sebastien@DietPi:~$

Seb-tech77 avatar Jan 22 '25 15:01 Seb-tech77

Try this:

apt install linux-headers-vendor-rk35xx
apt install zfs-dkms

MichaIng avatar Jan 22 '25 15:01 MichaIng

It's done, both orders have been executed successfully.

Done compiling kernel-headers tools (6.1.84-vendor-rk35xx). Done compiling kernel-headers tools (6.1.84-vendor-rk35xx). Armbian 'linux-headers-vendor-rk35xx' for '6.1.84-vendor-rk35xx': 'postinst' finishing. Lecture des listes de paquets... Fait Construction de l'arbre des dépendances... Fait Lecture des informations d'état... Fait zfs-dkms est déjà la version la plus récente (2.1.11-1+deb12u1). zfs-dkms passé en « installé manuellement ». 0 mis à jour, 0 nouvellement installés, 0 à enlever et 0 non mis à jour. Sebastien@DietPi:~$

Seb-tech77 avatar Jan 22 '25 15:01 Seb-tech77

So zfs-dkms was installed already, but the headers were likely missing before. Please try to reconfigure it to compile the kernel module:

dpkg-reconfigure zfs-dkms

MichaIng avatar Jan 22 '25 15:01 MichaIng

the command is finished but I get a segmentation error when I try to launch the ZFS module

Image

Seb-tech77 avatar Jan 22 '25 16:01 Seb-tech77

Can you paste the output of the module build? And:

modinfo zfs

MichaIng avatar Jan 22 '25 16:01 MichaIng

strange, I wanted to try a reboot but it doesn't want to start anymore, it's been trying to boot for more than 5min.

Image

Seb-tech77 avatar Jan 22 '25 16:01 Seb-tech77

No, it doesn't boot anymore, I can't even restore my backup now.

Seb-tech77 avatar Jan 22 '25 16:01 Seb-tech77

The netfilter-persistent service hangs forever. But before that, ifupdown-pre fails. It basically waits for udev to settle, before network connections can be reliably established, else the network adapters might not be ready yet. But ifup@eth0 etc do not depend on ifupdown-pre, so even if the latter fails, the former will run. But netfilter-persistent seems to aim for setting up firewall rules before networks are brought up, which makes sense, but hangs. This does not seem to be related to ZFS, as long as nothing it uses is on a ZFS filesystem, which hangs to mount of so.

Likely unrelated: You have cpufrequtils installed, which is not needed on DietPi, as dietpi-preboot applies the CPU governor and settings already, as can be seen some lines above. It does not hurt, but doubles the job. Also, the loadcpufreq service from this package fails for whatever reason, so that failure would at least be gone. You can change the CPU governor in dietpi-config performance options then, though there is almost never a reason to switch away from schedutil, IMO.

Why udev does not settle is another question, but the main problem is the netfilter service. You could disable the service by attacking the card/drive to another Linux system, then boot, and test netfilter-persistent start from console to see why it hangs.

MichaIng avatar Jan 23 '25 22:01 MichaIng

Hello, it's strange because net-filter has been installed for a while, it's only when I compiled the ZFS module that I had the problem. I know I don't see the connection either but apart from that I haven't done anything else. I've reset my system and I've managed to restore my backup so everything's fine, the system's working properly again. I'm going to make an image directly from my SD card and retest the commands.

For the cpu module, it must have been installed by openmediavault by default, but I use dietpi-config to adjust my cpu performance.

Seb-tech77 avatar Jan 24 '25 09:01 Seb-tech77

I'll try it on my Orange Pi 5 Max.

EDIT: So here is what I did:

root@DietPi:~# apt install linux-headers-vendor-rk35xx
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
...
Armbian 'linux-headers-vendor-rk35xx' for '6.1.84-vendor-rk35xx': 'postinst' finishing.
Processing triggers for libc-bin (2.36-9+deb12u9) ...
root@DietPi:~# apt install zfs-dkms
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
...

Then a confirmation prompt "Licenses of OpenZFS and Linux are incompatible" appears ...

...
Loading new zfs-2.1.11 DKMS files...
Building for 6.1.84-vendor-rk35xx
Building initial module for 6.1.84-vendor-rk35xx
Done.
...
zfs.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.1.84-vendor-rk35xx/updates/dkms/
...
depmod....
...
root@DietPi:~# modprobe zfs
Segmentation fault
root@DietPi:~# dmesg
...
[  882.789223] spl: loading out-of-tree module taints kernel.
[  882.797630] Unable to handle kernel paging request at virtual address 97fff80497ffe857
[  882.798376] Mem abort info:
[  882.798635]   ESR = 0x0000000096000004
[  882.798971]   EC = 0x25: DABT (current EL), IL = 32 bits
[  882.799453]   SET = 0, FnV = 0
[  882.799734]   EA = 0, S1PTW = 0
[  882.800027]   FSC = 0x04: level 0 translation fault
[  882.800462] Data abort info:
[  882.800750]   ISV = 0, ISS = 0x00000004
[  882.801098]   CM = 0, WnR = 0
[  882.801368] [97fff80497ffe857] address between user and kernel address ranges
[  882.802003] Internal error: Oops: 0000000096000004 [#1] SMP
[  882.802507] Modules linked in: spl(O+) btsdio bcmdhd goodix_ts rk805_pwrkey pwm_fan nvmem_rockchip_otp panfrost drm_shmem_helper gpu_sched rockchip_cpuinfo uio_pdrv_genirq uio fuse dm_mod ip_tables ipv6 r8169 adc_keys pwm_bl
[  882.804314] CPU: 2 PID: 33900 Comm: modprobe Tainted: G           O       6.1.84-vendor-rk35xx #1
[  882.805103] Hardware name: RK3588 OPi 5 Max (DT)
[  882.805516] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  882.806139] pc : mod_sysfs_setup+0x230/0x518
...

Yeah, definitely something wrong here. Not sure whether it is with this module in particular, or the headers 🤔.

There are also reports at Armbian, seems to be the Rockchip kernel: https://forum.armbian.com/topic/49378-kernel-oops-after-zfs-module-and-6184-vendor-rk35xx/

Trying backports:

root@DietPi:~# apt install -t bookworm-backports zfs-dkms

modprobe hangs now, need to reboot, but blacklisting zfs module to avoid boot issue. But issue remains with the other version as well. The spl module also hangs, so it is somewhat all results of that build.

Makes sense to test building another module, whether it is ZFS exclusively.

MichaIng avatar Jan 24 '25 12:01 MichaIng

Okay, thanks for testing on your side, if I understand correctly there's just waiting for an update of the rockchip kernel? Hoping it will be solved

Seb-tech77 avatar Jan 24 '25 13:01 Seb-tech77

Looks like it is the kernel headers, not the module/source itself:

root@DietPi:~# modprobe r8168
Segmentation fault

I'm triggering a new kernel build, just in case it has been fixed in the meantime.

MichaIng avatar Jan 24 '25 15:01 MichaIng

Are headers still broken with latest kernel upgrade?

MichaIng avatar Jun 02 '25 21:06 MichaIng

Yes they are. Tracking the issue here: #7412

MichaIng avatar Jun 22 '25 20:06 MichaIng

I can install and use zfs now on my Orange Pi 5. This is after I upgrade to Debian 13 Trixie. Last time I tried, I remember I was having issues. Just install linux-headers-vendor-rk35xx and zfs-dkms zfsutils-linux then follow the configuration screen

➜  ~ uname -r
6.1.115-vendor-rk35xx
➜  ~ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 13 (trixie)
Release:        13
Codename:       trixie
➜  ~ dpkg -l | grep linux-
ii  binutils-aarch64-linux-gnu         2.44-3                               arm64        GNU binary utilities, for aarch64-linux-gnu target
ii  cpp-14-aarch64-linux-gnu           14.2.0-19                            arm64        GNU C preprocessor for aarch64-linux-gnu
ii  cpp-aarch64-linux-gnu              4:14.2.0-1                           arm64        GNU C preprocessor (cpp) for the arm64 architecture
ii  gcc-14-aarch64-linux-gnu           14.2.0-19                            arm64        GNU C compiler for the aarch64-linux-gnu architecture
ii  gcc-aarch64-linux-gnu              4:14.2.0-1                           arm64        GNU C compiler for the arm64 architecture
ii  linux-base                         4.12                                 all          Linux image base package
ii  linux-dtb-vendor-rk35xx            25.11.0-trunk-dietpi1                arm64        Armbian Linux vendor DTBs in /boot/dtb-6.1.115-vendor-rk35xx
ii  linux-headers-6.12.43+deb13-arm64  6.12.43-1                            arm64        Header files for Linux 6.12.43+deb13-arm64
ii  linux-headers-6.12.43+deb13-common 6.12.43-1                            all          Common header files for Linux 6.12.43+deb13
ii  linux-headers-arm64                6.12.43-1                            arm64        Header files for Linux arm64 configuration (meta-package)
ii  linux-headers-vendor-rk35xx        25.11.0-trunk-dietpi1                arm64        Armbian Linux vendor headers 6.1.115-vendor-rk35xx
ii  linux-image-6.12.43+deb13-arm64    6.12.43-1                            arm64        Linux 6.12 for 64-bit ARMv8 machines (signed)
ii  linux-image-arm64                  6.12.43-1                            arm64        Linux for 64-bit ARMv8 machines (meta-package)
ii  linux-image-vendor-rk35xx          25.11.0-trunk-dietpi1                arm64        Armbian Linux vendor kernel image 6.1.115-vendor-rk35xx
ii  linux-kbuild-6.12.43+deb13         6.12.43-1                            arm64        Kbuild infrastructure for Linux 6.12.43+deb13
ii  linux-libc-dev                     6.12.43-1                            all          Linux support headers for userspace development
ii  linux-u-boot-orangepi5-vendor      25.08.0-trunk-dietpi1                arm64        Das U-Boot for orangepi5
➜  ~ lsmod | grep zfs
zfs                  3977216  1
spl                    90112  1 zfs

vuhuucuong avatar Sep 23 '25 13:09 vuhuucuong

That is great, will verify for some other modules ASAP in #7412.

MichaIng avatar Sep 23 '25 20:09 MichaIng