nestybox
Starting docker:dind works but immediatly creates a core dump wihtin the container as /core
With runc this does not happen:
docker run -it --rm --runtime runc --privileged --userns host docker:dind
With sysbox the core dump is created in /core within the container:
docker run -it --rm --runtime sysbox-runc docker:dind
I compared the console output of both commands and saw only these additional lines with sysbox-runc and no sign of the core dump:
[...]
mount: permission denied (are you root?)
Could not mount /sys/kernel/security.
AppArmor detection and --privileged mode might break.
[...]
WARN[2021-04-30T14:17:38.544203142Z] Not using native diff for overlay2, this may cause degraded performance for building images: running in a user namespace storage-driver=overlay2
[...]
Looking at the core dump i think it comes from a process called /check. But i am not a specialist for core dumps. But i often find this string within it:
strings core
CORE
CORE
check
/check
IGISCORE
CORE
ELIFCORE
/check
/check
CORE
FLINUX
LINUX
__kernel_vsyscall
__kernel_sigreturn
__kernel_rt_sigreturn
__vdso_gettimeofday
__vdso_time
__vdso_clock_gettime
__vdso_clock_gettime64
__vdso_clock_getres
linux-gate.so.1
LINUX_2.6
LINUX_2.5
Linux
Linux
5.10.0-0.bpo.5-amd64
B]wB
$[^_]
T[^_]
[^_]
\[^_]
d[^_]
d[^_]
<>9}
[^_]
T[^_]
<>9}
GCC: (Debian 8.3.0-6) 8.3.0
.shstrtab
.gnu.hash
.dynsym
.dynstr
.gnu.version
.gnu.version_d
.dynamic
.rodata
.note
.eh_frame_hdr
.eh_frame
.text
.altinstructions
.altinstr_replacement
.comment
v,M^i686
/check
HOSTNAME=b3aadbefad02
SHLVL=2
HOME=/root
container=docker
DIND_COMMIT=ed89041433a031cafc0a0f19cfe573c31688d377
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DOCKER_VERSION=20.10.6
DOCKER_TLS_CERTDIR=/certs
PWD=/
TMPDIR=/var/lib/docker/tmp
/check
Thanks Andreas; this is strange, I just played with the docker:dind image a few days ago and did not see any problems. Let me double check.
Question: apart from the core-dump inside the dind container, did you check if docker is running properly?
@nudgegoonies, which images were you running when you saw this? The latest patched sysbox-fs I provided as part of the other issue I'm working on, or is this with your original binaries? I would expect to see the same behavior either way, but just want to make sure that's the case.
I am not able to reproduce the core dump (don't see anything inside the /core directory). I am using sysbox-ce v0.3.0.
I do see the inner Docker generating these warnings as you mentioned:
mount: permission denied (are you root?)
Could not mount /sys/kernel/security.
AppArmor detection and --privileged mode might break.
The above are because the Sysbox container is rootless, so mounting /sys/kernel/security is not allowed (for apparmor). This will not be a problem however because the Docker at host level will have inserted the apparmor profile into the kernel already.
WARN[2021-04-30T14:17:38.544203142Z] Not using native diff for overlay2, this may cause degraded performance for building images: running in a user namespace storage-driver=overlay2
This warning I need to investigate, but it's not related to the core dump you mentioned in any way.
My host Docker is at version 20.10.5; the Docker running inside the docker:dind container is at version 20.10.6.
I am able to run inner containers without problem.
Question: apart from the core-dump inside the dind container, did you check if docker is running properly?
Yes. software inside the container runs.
which images were you running when you saw this? The latest patched sysbox-fs I provided as part of the other issue I'm working on, or is this with your original binaries? I would expect to see the same behavior either way, but just want to make sure that's the case.
I used the master build with the patched sysbox-fs:
sysbox-runc
edition: Community Edition (CE)
version: 0.3.0
commit: df952e5276cb6e705e0be331e9a9fe88f372eab8
built at: Thu Apr 29 08:08:51 UTC 2021
oci-specs: 1.0.2-dev
sysbox-mgr
edition: Community Edition (CE)
version: 0.3.0
commit: 5f2be318c7559cd04074016614b3aff47d61a6e0
built at: Thu Apr 29 08:09:01 UTC 2021
sysbox-fs
edition: Community Edition (CE)
version: 0.3.0
commit: 85d065dc1cc0027d1ea7827396cb11818e36779f-dirty
built at: Thu Apr 29 08:08:58 UTC 2021
I am running this version on together with docker-ce package 5:20.10.6~3-0~debian-buster and the backport kernel package 5.10.0-0.bpo.5-amd64 running kernel 5.10.24-1~bpo10+1.
I can also start inner containers like debian:buster without problem, only one error appears that does not has an effect on the running container:
time="2021-05-04T06:51:01.098750481Z" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e4185431e7d400142a828b1e41b7e7f081dcbd5114a3ae533ca0cc2176517d81 pid=845
INFO[2021-05-04T06:51:08.880504057Z] shim disconnected id=e4185431e7d400142a828b1e41b7e7f081dcbd5114a3ae533ca0cc2176517d81
INFO[2021-05-04T06:51:08.880551328Z] ignoring event container=e4185431e7d400142a828b1e41b7e7f081dcbd5114a3ae533ca0cc2176517d81 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
ERRO[2021-05-04T06:51:08.880563211Z] copy shim log error="read /proc/self/fd/12: file already closed"
Hi @nudgegoonies,
Still not able to repro on my side:
$ docker run -it --rm --runtime sysbox-runc docker:dind
runs without problem, no core file gets generated under /core inside the container. The inner Docker is version 20.10.8.
In my case, I am on a Ubuntu-Focal host with kernel 5.4:
$ uname -a
Linux focal 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Question: what are the exact steps you use to reproduce the problem? Without a repro on our side it will be difficult for us to fix this.
Thanks as always for using Sysbox and reporting all issues you find.
Hi @ctalledo Thank you for the answer. We are on Debian 10 with docker 20.10.8 and userns. I start the docker dind 20.10.8 with:
docker run -it --rm --name test1234 docker:dind
In the syslog i can find this:
[3900047.222337] overlayfs: unrecognized mount option "userxattr" or missing value
[3900047.242895] new mount options do not match the existing superblock, will be ignored
This is a file on the core dump:
❯ file core
core: ELF 32-bit LSB core file, Intel 80386, version 1 (SYSV), SVR4-style, from '/check', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/check', platform: 'i686'
Strange, that this is an 32-bit executable that fails because this is an amd64 Debian without multiarch and even within the dind container i cannot find a check application. Maybe somthing docker mounts internally there at the start?
Hi @nudgegoonies,
Sounds like this is somehow related to this commit in the 5.11 kernel:
OVERLAYFS
(FEATURED) Unprivileged mounts commit
Add the -o userxattr mount option forces overlayfs to use the "user.overlay." xattr namespace instead of "trusted.overlay.". This is useful for unprivileged mounting of overlayfs commit.
The following error:
overlayfs: unrecognized mount option "userxattr" or missing value
suggests that the kernel in your machine does yet support the userxattr overlayfs option, yet some process (I am guessing Docker or some other) is using it.
Does the Docker log show anything related to this?
We are using the Backport Kernel 5.10.40-1~bpo10+1 for Debian 10. This is the Kernel from Debian 11 so updating to Debian 11 will not bring us the latest Kernel because there is no Backport Kernel in Debian 11 yet.
There is nothing suspicious in the host docker-log, containerd-log, sysbox-fs and sysbox-mgr log. But this message in the dind-log:
WARN[2021-09-10T08:14:48.443144973Z] Not using native diff for overlay2, this may cause degraded performance for building images: running in a user namespace storage-driver=overlay2
Hi @nudgegoonies ,
In your Debian 10 host with the Backport Kernel 5.10.40-1~bpo10+1, what do you see when you type uname -a?
As I mentioned, I suspect Docker thinks the overlayfs in your kernel supports the userxattr mount option, but it does not (as evidenced by the syslog message you reported).
Also: you mentioned this command generates the error in the syslog:
docker run -it --rm --name test1234 docker:dind
In this command running at host level? If so, if sysbox-runc configured as Docker default runtime?
Or is this command running inside a Sysbox container?
Hi @ctalledo This is the full uname -a output:
Linux mam-gitlabci-bs8001 5.10.0-0.bpo.7-amd64 #1 SMP Debian 5.10.40-1~bpo10+1 (2021-06-04) x86_64 GNU/Linux
I ran this command at host level and sysbox-runc is configured as Docker default runtime.
Sounds like the answer is here:
https://github.com/moby/moby/blob/5176095455642c30642efacf6f35afc7c6dede92/daemon/graphdriver/overlayutils/userxattr.go#L34
On kernels < 5.11 (as in your case), it looks like Docker performs an "overlayfs test" to see if the userxattr option is present. That test is likely causing the syslog message you reported (overlayfs: unrecognized mount option "userxattr" or missing value).
So this message is a red-herring, meaning that it does not point to any problem per-se (it's expected as part of the test).
Back to the issue originally reported (core-dump within the dind container), I was never able to repro that, but I don't have the same setup that you do (Debian 10 with docker 20.10.8 and userns). Any ideas what process generated that core dump?
Could it be, that the /check program is something injected by docker?
file core
core: ELF 32-bit LSB core file, Intel 80386, version 1 (SYSV), SVR4-style, from '/check', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/check', platform: 'i686'
Its an i686 application.
Hi @nudgegoonies,
Could it be, that the /check program is something injected by docker?
It appears so, I see references to it in the Docker buildkit repo.
I am no expert on buildkit, but I know it supports multi-arch builds and it appears the code that fails is a setup check related to this.
I also know that buildkit multi-arch builds with QEMU don't yet work inside a Sysbox container (it's something we are enabling for the Sysbox-EE edition right now).
Are you using buildkit inside the Sysbox container?
Not explicit. Maybe implicit when this is the default. I check this.
No. I disabled it within daemon.json, the daemon starts, but also generates a core file:
{ "features": { "buildkit": false } }
No. I disabled it within daemon.json, the daemon starts, but also generates a core file
I see ... not sure how to help next; when I looked at the Moby repo the buildkit section was the only part of the code I found a reference to the "/check" program.
Hi @nudgegoonies; any update on this issue? It seems stale and we were never able to repro on Nestybox's side. If no updates, we can close and re-open if it shows up again.
Hi @ctalledo I will check this issue and https://github.com/nestybox/sysbox/issues/414 with sysbox 0.5.0 next week.
Still the case with sysbox 0.5.0 on docker 20.10.14 with docker-dind 20.10.7 and with docker-dind 20.10.14
Hi @nudgegoonies, thanks for the update; I'll see if I can repro with v0.5.0 too, though I doubt it given that I was not able to repro before. We've also not seen other reports of this issue, some it's something specific to your setup.