Unable to debug crashing distroless container

[rickyson96]

Hi! Thanks for your work!

I stumbled on this tool because I wish to debug crashed distroless container, but unable to do so.

To replicate this issue, I have tried the following

docker run \
        --name my-distroless gcr.io/distroless/nodejs \
        -e 'console.log("Done")'

And tried to copy and start the container with this command, but it failed.

$ debug-ctr debug --image=docker.io/alpine --target=my-distroless --copy-to=crashing-container-copy --entrypoint="/.debugger/sleep" --cmd="265d"
...
Error: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/.debugger/sleep": stat /.debugger/sleep: no such file or directory: unknown
...

I have tried to see what's going on inside the container (and thankfully I somehow used nixery image, and it replicates there) with these commands.

$ docker run --name crashing-container nixery.dev/shell /bin/sh -c "false"
...

$ debug-ctr debug --image=docker.io/alpine --target=crashing-container --copy-to=crashing-container-copy --entrypoint="sleep" --cmd="265d"
...

$ docker exec -it crashing-container-copy sh

sh# ls -l /.debugger
...

Somehow, the resulting binaries that are symlinked has 0 bytes, and /bin/busybox was not found in the resulting image.

I also tried to debug into the now running copier container, but also faced with errors

$ go run main.go debug --target=crashing-container-copy
Error: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "proc" to rootfs at "/proc": mount proc:/proc (via /proc/self/fd/6), flags: 0xe: operation not permitted: unknown

However, it was able to create the debugger succesfully, if the crashing container was based on busybox or alpine.