sysbox icon indicating copy to clipboard operation
sysbox copied to clipboard
verified publisher icon nestybox

"OCI runtime attempted to invoke a command that was not found" error from Podman in sysbox system container

Open DekusDenial opened this issue 2 years ago • 4 comments

I was trying to follow this post (https://www.redhat.com/sysadmin/podman-inside-kubernetes), and hope to see if I can get Podman working without privileged mode in a K8s pod running Sysbox system container. I understand there are already existing issues regarding Podman integration into Sysbox so I am here to provide more info:

  • https://github.com/nestybox/sysbox/issues/100
  • https://github.com/nestybox/sysbox/issues/128

At first I was getting error about /dev/fuse not found and I am aware of the limitation in Sysbox related to this. I knew I need to have it mounted from host , so I was able to workaround by using the io.kubernetes.cri-o.Devices annotation instead of using the kubelet device plugin as mentioned from the post, and made /dev/fuse with permission 0666. However, I then got the following error when trying to start a container via Podman:

sh-5.2# podman  run --rm hello-world
WARN[0000] Found incomplete layer "942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c", deleting it
Error: crun: make `/var/lib/containers/storage/overlay/edd2911b92f517336aa38932e54809852d2f9c6ab718c7df7a3bfb9bdf587b39/merged` private: No such file or directory: OCI runtime attempted to invoke a command that was not found

But, if I create the pod/container with default runc runtime in CRI-O instead, I was able to get pass that error. Of course, the --privileged flag will not work in this case.

  • AWS, Amazon Linux 2, kernel: 5.15.104-63.140.amzn2.x86_64, EKS v1.26
  • crio version: 1.26.1
  • sysbox version: 0.6.1
  • pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: podman
  annotations:
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
    io.kubernetes.cri-o.Devices: /dev/fuse
spec:
  runtimeClassName: sysbox-runc
  containers:
  - name: podman
    image: quay.io/podman/stable
    command: ["sh", "-c", "sleep inf"]
    securityContext:
      capabilities:
        add:
          - "SYS_ADMIN"
          - "MKNOD"
          - "SYS_CHROOT"
          - "SETFCAP"
  restartPolicy: Never
  • podman debugged error
sh-5.2# podman --log-level debug run --rm hello-world
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --log-level debug run --rm hello-world)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: imagestore=/var/lib/shared
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=overlayfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 7
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image hello-world (policy: missing)
DEBU[0000] Looking up image "hello-world" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Found image "hello-world" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "hello-world" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76)
WARN[0000] Found incomplete layer "85b6b1ba6b6f2912b269ad09021115d1f378ee3c1f1de3b9c7ebfaba3ff50208", deleting it
DEBU[0000] exporting opaque data as blob "sha256:54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Looking up image "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Found image "quay.io/podman/hello:latest" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "quay.io/podman/hello:latest" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76)
DEBU[0000] exporting opaque data as blob "sha256:54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Looking up image "hello-world" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Found image "hello-world" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "hello-world" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76)
DEBU[0000] exporting opaque data as blob "sha256:54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Inspecting image 54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76
DEBU[0000] exporting opaque data as blob "sha256:54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Inspecting image 54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76
DEBU[0000] Inspecting image 54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76
DEBU[0000] Inspecting image 54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76
DEBU[0000] Inspecting image 54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76
DEBU[0000] using systemd mode: false
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json"
INFO[0000] Sysctl net.ipv4.ping_group_range=0 0 ignored in containers.conf, since Network Namespace set to host
DEBU[0000] Allocated lock 0 for container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] exporting opaque data as blob "sha256:54c80734fe405a23783a26881d74c5842f6b047f021b029c0b672565101fef76"
DEBU[0000] Created container "a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807"
DEBU[0000] Container "a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807" has work directory "/var/lib/containers/storage/overlay-containers/a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807/userdata"
DEBU[0000] Container "a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807" has run directory "/run/containers/storage/overlay-containers/a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807/userdata"
DEBU[0000] Not attaching to stdin
INFO[0000] Received shutdown.Stop(), terminating!        PID=242
DEBU[0000] Enabling signal proxying
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/U5CGCP6ZNJKDVUM7RPSUQBVITU,upperdir=/var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/diff,workdir=/var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/work,nodev,fsync=0,volatile
DEBU[0000] Mounted container "a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807" at "/var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged"
DEBU[0000] Created root filesystem for container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 at /var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged
DEBU[0000] Modifying container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 /etc/passwd
DEBU[0000] Modifying container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 /etc/group
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Workdir "/" resolved to host path "/var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged"
DEBU[0000] Created OCI spec for container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 at /var/lib/containers/storage/overlay-containers/a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] Running with no Cgroups
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 -u a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807/userdata -p /run/containers/storage/overlay-containers/a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807/userdata/pidfile -n sad_blackwell --exit-dir /run/libpod/exits --full-attach -l k8s-file:/var/lib/containers/storage/overlay-containers/a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807/userdata/ctr.log --log-level debug --syslog --runtime-arg --cgroup-manager --runtime-arg disabled --conmon-pidfile /run/containers/storage/overlay-containers/a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.imagestore=/var/lib/shared --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,fsync=0 --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1
DEBU[0000] Cleaning up container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Error unmounting /var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged with fusermount3 - exit status 1
DEBU[0000] Failed to remove mountpoint 942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c overlay: /var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged - device or resource busy
DEBU[0000] Unmounted container "a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807"
DEBU[0000] Removing container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807
DEBU[0000] Cleaning up container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 storage is already unmounted, skipping...
DEBU[0000] Removing all exec sessions for container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807
DEBU[0000] Container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 storage is already unmounted, skipping...
DEBU[0000] Failed to delete container "a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807": 1 error occurred:
	* unlinkat /var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c: directory not empty

DEBU[0000] unable to remove container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 after failing to start and attach to it: removing container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 root filesystem: 1 error occurred:
	* unlinkat /var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c: directory not empty

DEBU[0000] ExitCode msg: "crun: make `/var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged` private: no such file or directory: oci runtime attempted to invoke a command that was not found"
Error: crun: make `/var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged` private: No such file or directory: OCI runtime attempted to invoke a command that was not found
DEBU[0000] Shutting down engines

DekusDenial avatar Apr 21 '23 01:04 DekusDenial

@rodnymolina @ctalledo any thought?

DekusDenial avatar Apr 26 '23 01:04 DekusDenial

Hi @DekusDenial, thanks for trying and documenting this effort.

There are a couple of issues to address here before we can support what you are attempting to do:

  1. First of all, we need to support rootful podman within a sysbox container, which technically speaking isn't a hard thing to do taking into account where we left off last time we worked on this area -- rootless podman within sysbox would be a totally different story, but I see little value in doing that once the wrapping sysbox container is rootless itself.

  2. Extend Sysbox and sysbox-k8s-deploy daemonset support to AL2. As before, this is not rocket science either, but would require cycles that we currently don't have.

In short, it would be difficult for me to give you an ETA for this, but if you are interested, we could help you make these enhancements by your own. Let me know if that's the case.

rodnymolina avatar Apr 28 '23 18:04 rodnymolina

@rodnymolina thank you for the update. Don’t worry about 2) at all. We are more interested in 1) for rootful in particular and would like help in that even if it means we have to make some changes on our side as we really want sysbox to be the backing runtime for our workload ATM.

DekusDenial avatar Apr 28 '23 19:04 DekusDenial

@rodnymolina @ctalledo does this sound like something easy to do or you think this needs a second thought? If you point me to where I can make such enhancements, it’d be nice.

DekusDenial avatar May 16 '23 08:05 DekusDenial