sysbox icon indicating copy to clipboard operation
sysbox copied to clipboard

sysbox-fs triggers mount(2) denial (EACCES) on Ubuntu 25.04

Open vanackere opened this issue 10 months ago • 7 comments

Environment:

  • Sysbox Version: 0.6.7 (and latest from master)
  • Host OS: Ubuntu 25.04 (Mantic Minotaur)
  • Kernel Version: 6.14.0-15-generic
  • Docker Version: 27.5.1 (27.5.1-0ubuntu3)

Problem Summary

Sysbox containers fail to start on Ubuntu 25.04 due to a Permission denied error during FUSE mount setup.

> docker run --runtime=sysbox-runc -it --rm alpine:latest
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: failed to pre-register with sysbox-fs: failed to pre-register container with sysbox-fs: rpc error: code = DeadlineExceeded desc = context deadline exceeded: unknown.

sysbox-fs log reports:

> journalctl -u sysbox-fs
Jun 11 15:23:28 quartz sysbox-fs[356624]: time="2025-06-11 15:23:28" level=debug msg="Container pre-registration started: id =>
Jun 11 15:23:28 quartz sysbox-fs[356624]: 2025/06/11 15:23:28 mount helper error: fusermount: mount failed: Permission denied
Jun 11 15:23:28 quartz sysbox-fs[356624]: time="2025-06-11 15:23:28" level=error msg="fusermount: exit status 1"

Debugging: Seccomp / Apparmor issues were ruled out since I get the same failure with

docker run --runtime=sysbox-runc --security-opt apparmor=unconfined --security-opt seccomp=unconfined -it --rm alpine:latest

By using strace on the fusermount helper process launched by sysbox-fs, I get this:

1397518 execve("/usr/bin/fusermount", ["fusermount", "-o", "fsname=sysboxfs,allow_other,defa"..., "--", "/var/lib/sysboxfs/87d2b980403e92"...], 0x7ffc17bf6c68 /* 18 vars */) = 0
1397518 brk(NULL)                       = 0x63359104f000
1397518 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7c51f3eeb000
1397518 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
1397518 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
1397518 fstat(4, {st_mode=S_IFREG|0644, st_size=91117, ...}) = 0
1397518 mmap(NULL, 91117, PROT_READ, MAP_PRIVATE, 4, 0) = 0x7c51f3ed4000
1397518 close(4)                        = 0
1397518 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 4
1397518 read(4, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\245\2\0\0\0\0\0"..., 832) = 832
1397518 pread64(4, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 840, 64) = 840
1397518 fstat(4, {st_mode=S_IFREG|0755, st_size=2182752, ...}) = 0
1397518 pread64(4, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 840, 64) = 840
1397518 mmap(NULL, 2227832, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0x7c51f3c00000
1397518 mmap(0x7c51f3c28000, 1662976, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x28000) = 0x7c51f3c28000
1397518 mmap(0x7c51f3dbe000, 323584, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x1be000) = 0x7c51f3dbe000
1397518 mmap(0x7c51f3e0d000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x20c000) = 0x7c51f3e0d000
1397518 mmap(0x7c51f3e13000, 52856, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7c51f3e13000
1397518 close(4)                        = 0
1397518 mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7c51f3ed1000
1397518 arch_prctl(ARCH_SET_FS, 0x7c51f3ed1740) = 0
1397518 set_tid_address(0x7c51f3ed1a10) = 1397518
1397518 set_robust_list(0x7c51f3ed1a20, 24) = 0
1397518 rseq(0x7c51f3ed1680, 0x20, 0, 0x53053053) = 0
1397518 mprotect(0x7c51f3e0d000, 16384, PROT_READ) = 0
1397518 mprotect(0x633554620000, 4096, PROT_READ) = 0
1397518 mprotect(0x7c51f3f2d000, 8192, PROT_READ) = 0
1397518 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
1397518 munmap(0x7c51f3ed4000, 91117)   = 0
1397518 getrandom("\x8c\x6f\xaf\x08\x83\x0f\x3e\x7a", 8, GRND_NONBLOCK) = 8
1397518 brk(NULL)                       = 0x63359104f000
1397518 brk(0x633591070000)             = 0x633591070000
1397518 getuid()                        = 0
1397518 readlink("/var", 0x7ffe29e8e780, 1023) = -1 EINVAL (Invalid argument)
1397518 readlink("/var/lib", 0x7ffe29e8e780, 1023) = -1 EINVAL (Invalid argument)
1397518 readlink("/var/lib/sysboxfs", 0x7ffe29e8e780, 1023) = -1 EINVAL (Invalid argument)
1397518 chdir("/")                      = 0
1397518 getuid()                        = 0
1397518 umask(033)                      = 022
1397518 getuid()                        = 0
1397518 openat(AT_FDCWD, "/dev/fuse", O_RDWR) = 4
1397518 getuid()                        = 0
1397518 getuid()                        = 0
1397518 openat(AT_FDCWD, "/etc/fuse.conf", O_RDONLY) = 5
1397518 fstat(5, {st_mode=S_IFREG|0644, st_size=694, ...}) = 0
1397518 read(5, "# The file /etc/fuse.conf allows"..., 4096) = 694
1397518 read(5, "", 4096)               = 0
1397518 close(5)                        = 0
1397518 getuid()                        = 0
1397518 newfstatat(AT_FDCWD, "/var/lib/sysboxfs/87d2b980403e925d3c864a064092b0e912f4a5a37abf04436d9d3e8926d29367", {st_mode=S_IFDIR|0600, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
1397518 getuid()                        = 0
1397518 getuid()                        = 0
1397518 getuid()                        = 0
1397518 getuid()                        = 0
1397518 getuid()                        = 0
1397518 getgid()                        = 0
1397518 getuid()                        = 0
1397518 mount("sysboxfs", "/var/lib/sysboxfs/87d2b980403e925d3c864a064092b0e912f4a5a37abf04436d9d3e8926d29367", "fuse", MS_NOSUID|MS_NODEV, "allow_other,default_permissions,"...) = -1 EACCES (Permission denied)
1397518 write(2, "fusermount: mount failed: Permis"..., 44) = 44
1397518 close(4)                        = 0
1397518 exit_group(1)                   = ?
1397518 +++ exited with 1 +++

vanackere avatar Jun 13 '25 06:06 vanackere

Thanks @vanackere for reporting, I'll take a look ASAP.

ctalledo avatar Jun 18 '25 01:06 ctalledo

I was able to reproduce easily on a Ubuntu-25.04 host; I'll investigate what's going on.

ctalledo avatar Jun 20 '25 00:06 ctalledo

Hello !

Any news about this issue ? Do you think it will be easy to fix ? (Asking because depending on the response I'll work in a VM to make use of sysbox in the meantime ;-))

vanackere avatar Jul 01 '25 12:07 vanackere

@ctalledo any help on this

abhi4u1947 avatar Jul 10 '25 10:07 abhi4u1947

@vanackere, @abhi4u1947, no update yet but I am looking at it right now.

ctalledo avatar Jul 11 '25 21:07 ctalledo

OK, got to the bottom of this: In Ubuntu 25.04 (Plucky), an apparmor profile for the fusermount3 utility has been added under /etc/apparmor.d/fusermount3, and it's blocking Sysbox from creating filesystem-in-userspace (FUSE) mounts. Found this by looking at the kernel ring buffer as follows:

$ sudo dmesg | grep -i sysbox
[ 4404.088717] audit: type=1400 audit(1752516873.749:187): apparmor="DENIED" operation="mount" class="mount" info="failed mntpnt match" error=-13 profile="fusermount3" name="/var/lib/sysboxfs/635b361d38a587b33b5b7f2f36e67248ff47c5295d69b86275e52ca7f29b3293/"

In prior versions of Ubuntu (e.g., Ubuntu 24.04) no such profile existed so Sysbox was able to create these mounts without problem.

Solution

We need to update the Sysbox installation script to automatically edit /etc/apparmor.d/fusermount3 such that Sysbox is given permission to perform FUSE mounts under /var/lib/sysboxfs. This will come in the next release of Sysbox (v0.6.8).

Temporary Work-Around

In the meantime, edit the /etc/apparmor.d/fusermount3 file and add the following entries to it:

profile fusermount3 /usr/bin/fusermount3 {
  ...
  mount fstype=@{fuse_types} options=(nosuid,nodev,rw) -> /var/lib/sysboxfs/**/,
  mount fstype=@{fuse_types} options=(nosuid,nodev,ro) -> /var/lib/sysboxfs/**/,
  umount /var/lib/sysboxfs/**/,
  ...
}

You need sudo privileges to edit that file.

Once you've edited it, ask apparmor to reload it with:

sudo apparmor_parser -r /etc/apparmor.d/fusermount3

Also, restart Sysbox:

sudo systemctl restart sysbox

After this you should be able to use Sysbox on Ubuntu-25.04 as usual.

Hope that helps!

ctalledo avatar Jul 15 '25 00:07 ctalledo

In the meantime, edit the /etc/apparmor.d/fusermount3 file and add the following entries to it:

Alternatively, you can edit the "local" version of this file under /etc/apparmor.d/local/fusermount3 and simply add these 3 lines in it:

# allow Sysbox to create fuse mounts
mount fstype=@{fuse_types} options=(nosuid,nodev,rw) -> /var/lib/sysboxfs/**/,
mount fstype=@{fuse_types} options=(nosuid,nodev,ro) -> /var/lib/sysboxfs/**/,
umount /var/lib/sysboxfs/**/,

That is easier and more portable than editing the parent file /etc/apparmor.d/fusermount3.

Then follow that with:

sudo apparmor_parser -r /etc/apparmor.d/fusermount3
sudo systemctl restart sysbox

ctalledo avatar Jul 15 '25 00:07 ctalledo