nestybox
2020/10/21/gitlab-dind
Securing GitLab CI pipelines with Sysbox | Nestybox Blog Site
Article describing how to improve security of GitLab using Sysbox.
https://blog.nestybox.com/2020/10/21/gitlab-dind.html
GitLab appears to be waiting for someone to make a merge request fixing the GitLab runner ignores the "runtime" configuration for service containers issue in the GitLab runner: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27168
I'm eager to see this fixed; having sysbox work "out of the box" with GitLab is really exciting. Thank you for your work and for sharing this information!
We are currently working on integrating Kubernetes + Sysbox, to make it possible for K8s to deploy pods that are capable of running things like systemd, Docker, and K8s inside the pod itself, and do so securely (with rootless containers). We expect to have this very soon (weeks).
Once we have this, then using the GitLab K8s executor should be fairly straight-forward, assuming we can tell the GitLab K8s executor to deploy the pod using the Sysbox "runtime class". I would need to take a closer look as the devil is in the details.
I found a workaround to make the setup from the “Setups that (currently) don’t work” section work. 🙂
The trick is to run the GitLab runner Docker container using the traditional “runc” runtime, but have the Docker containers spawned by the GitLab runner use the “sysbox-runc” runtime.
While we have to make “sysbox-runc” the default runtime used by Docker (because setting the runtime used by the GitLab runner does not work reliably as described in the article), we can override the runtime used for the GitLab runner itself.
This can be done by simply adding “--runtime runc” when creating the container. For example:
docker run -d -v /path/to/gitlab-runner-config:/etc/gitlab-runner --runtime runc gitlab/gitlab-runner:alpine-v13.10.0
Thanks @smarsching, that's clever! Glad you found a way to do it since it's more convenient to use the containerized gitlab runner.
I've decided to ping subscribed people on here as well so that anyone interested can enable notifications on the merge request
I recently created a merge request to gitlab running resolving the issue mentioned above, its quite a simple change: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3063
I recently created a merge request to gitlab running resolving the issue mentioned above, its quite a simple change: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3063
Very kind of you @networkException! (I suspected it was a simple change but never got the chance to do it and test it ... thanks)
Thanks @networkException for bringing this to a resolution, much appreciated!
Once we have this, then using the GitLab K8s executor should be fairly straight-forward, assuming we can tell the GitLab K8s executor to deploy the pod using the Sysbox "runtime class". I would need to take a closer look as the devil is in the details.
I believe that this GitLab MR would need to be merge to do that: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/2326 Can you please confirm my understanding?
I believe that this GitLab MR would need to be merge to do that: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/2326 Can you please confirm my understanding?
Thanks @candrews ... looks like that MR is not moving fast unfortunately ...
Having some issue with GitLab Runner & Docker in a System Container setup.
runner config:
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "my-docker-runner"
url = "https://git.me.asdf/"
token = "TOKEN"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:20.10.12"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache", "/var/lib/docker"]
shm_size = 0
runtime = "sysbox-runc"
CI reported
Using Docker executor with image docker:20.10.12 ...
ERROR: Preparation failed: adding cache volume: set volume permissions: running permission container "33f2c95b8eacf22bba2f8480ed1e947afacb370bc909cb527b8dfc54c550b455" for volume "runner-y5eqdgvc-project-7-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70": starting permission container: Error response from daemon: cgroups: cgroup mountpoint does not exist: unknown (linux_set.go:100:2s)
sysbox running, /etc/docker/daemon.json
{
"default-runtime": "sysbox-runc",
"runtimes": {
"sysbox-runc": {
"path": "/usr/bin/sysbox-runc",
"runtimeArgs": ["--no-kernel-check"]
}
}
}
Inside the docker I got
root@7022cd7fae8d:/# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 216 4 ? Ss 01:20 0:00 /usr/bin/dumb-init /entrypoint run --user=gitlab-runner --working-directory=/home/gitlab-runner
root 52 0.6 0.0 140640 16280 ? Ssl 01:20 0:02 gitlab-runner run --user=gitlab-runner --working-directory=/home/gitlab-runner
root 54 1.8 0.4 2190480 71996 ? Sl 01:20 0:07 dockerd
root 86 0.4 0.1 1634128 24136 ? Ssl 01:20 0:01 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
root 605 0.0 0.0 410112 9224 ? Ssl 01:20 0:00 runc init
root 834 0.0 0.0 410112 9136 ? Ssl 01:20 0:00 runc init
root 1046 0.0 0.0 410112 9264 ? Ssl 01:20 0:00 runc init
root 1268 0.0 0.0 410112 10684 ? Ssl 01:30 0:00 runc init
root 1482 0.0 0.0 410112 10760 ? Ssl 01:30 0:00 runc init
root 1696 0.0 0.0 410112 10616 ? Ssl 01:31 0:00 runc init
root 1800 0.0 0.0 4252 3376 pts/0 Ss 01:37 0:00 bash
root 1808 0.0 0.0 5896 2876 pts/0 R+ 01:37 0:00 ps -aux
and
root@7022cd7fae8d:/var/log# cat dockerd.log
time="2022-02-28T01:20:19.818754521Z" level=info msg="Starting up"
time="2022-02-28T01:20:19.859157920Z" level=info msg="libcontainerd: started new containerd process" pid=86
time="2022-02-28T01:20:19.859220455Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2022-02-28T01:20:19.859243557Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2022-02-28T01:20:19.859281116Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
time="2022-02-28T01:20:19.859311612Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-02-28T01:20:20.246443014Z" level=info msg="starting containerd" revision=8fba4e9a7d01810a393d5d25a3621dc101981175 version=1.3.7
time="2022-02-28T01:20:20.274563494Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2022-02-28T01:20:20.274797787Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
time="2022-02-28T01:20:20.275445940Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2022-02-28T01:20:20.275491508Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2022-02-28T01:20:20.275518269Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2022-02-28T01:20:20.275639513Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="modprobe aufs failed: \"\": exec: \"modprobe\": executable file not found in $PATH: skip plugin" type=io.containerd.snapshotter.v1
time="2022-02-28T01:20:20.275677991Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2022-02-28T01:20:20.275757916Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2022-02-28T01:20:20.275972875Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2022-02-28T01:20:20.276260547Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2022-02-28T01:20:20.276298811Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2022-02-28T01:20:20.276374799Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2022-02-28T01:20:20.276415778Z" level=info msg="metadata content store policy set" policy=shared
time="2022-02-28T01:20:20.358517881Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2022-02-28T01:20:20.358570615Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2022-02-28T01:20:20.358638176Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2022-02-28T01:20:20.358672303Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2022-02-28T01:20:20.358698383Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2022-02-28T01:20:20.358724040Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2022-02-28T01:20:20.358762270Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
time="2022-02-28T01:20:20.358793181Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2022-02-28T01:20:20.358833557Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2022-02-28T01:20:20.358863334Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2022-02-28T01:20:20.359072826Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2022-02-28T01:20:20.359263802Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2022-02-28T01:20:20.359703431Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2022-02-28T01:20:20.359762373Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2022-02-28T01:20:20.359847512Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2022-02-28T01:20:20.359882761Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2022-02-28T01:20:20.359908481Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2022-02-28T01:20:20.359932453Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2022-02-28T01:20:20.359955078Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2022-02-28T01:20:20.359981907Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2022-02-28T01:20:20.360006443Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2022-02-28T01:20:20.360069419Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2022-02-28T01:20:20.360114914Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2022-02-28T01:20:20.360302141Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2022-02-28T01:20:20.360341458Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2022-02-28T01:20:20.360367243Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2022-02-28T01:20:20.360388960Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2022-02-28T01:20:20.361831454Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
time="2022-02-28T01:20:20.362178600Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
time="2022-02-28T01:20:20.362454885Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
time="2022-02-28T01:20:20.362501968Z" level=info msg="containerd successfully booted in 0.117422s"
time="2022-02-28T01:20:20.488434843Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2022-02-28T01:20:20.489927253Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2022-02-28T01:20:20.492620071Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
time="2022-02-28T01:20:20.492699880Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-02-28T01:20:20.494356705Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2022-02-28T01:20:20.494393053Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2022-02-28T01:20:20.494421326Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
time="2022-02-28T01:20:20.494465278Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-02-28T01:20:20.723517826Z" level=warning msg="Your kernel does not support swap memory limit"
time="2022-02-28T01:20:20.723586602Z" level=warning msg="Your kernel does not support memory reservation"
time="2022-02-28T01:20:20.723610944Z" level=warning msg="Your kernel does not support oom control"
time="2022-02-28T01:20:20.723629769Z" level=warning msg="Your kernel does not support memory swappiness"
time="2022-02-28T01:20:20.723647351Z" level=warning msg="Your kernel does not support kernel memory limit"
time="2022-02-28T01:20:20.723664270Z" level=warning msg="Your kernel does not support kernel memory TCP limit"
time="2022-02-28T01:20:20.723680605Z" level=warning msg="Your kernel does not support cgroup cpu shares"
time="2022-02-28T01:20:20.723696804Z" level=warning msg="Your kernel does not support cgroup cfs period"
time="2022-02-28T01:20:20.723713182Z" level=warning msg="Your kernel does not support cgroup cfs quotas"
time="2022-02-28T01:20:20.723729778Z" level=warning msg="Your kernel does not support cgroup rt period"
time="2022-02-28T01:20:20.723746838Z" level=warning msg="Your kernel does not support cgroup rt runtime"
time="2022-02-28T01:20:20.723763458Z" level=warning msg="Unable to find blkio cgroup in mounts"
time="2022-02-28T01:20:20.724088492Z" level=info msg="Loading containers: start."
time="2022-02-28T01:20:21.593906973Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address"
time="2022-02-28T01:20:21.952929712Z" level=info msg="Loading containers: done."
time="2022-02-28T01:20:22.166973468Z" level=info msg="Docker daemon" commit=4484c46d9d graphdriver(s)=btrfs version=19.03.13
time="2022-02-28T01:20:22.167229640Z" level=info msg="Daemon has completed initialization"
time="2022-02-28T01:20:22.535504251Z" level=info msg="API listen on /var/run/docker.sock"
time="2022-02-28T01:20:22.647414125Z" level=error msg="Handler for POST /v1.25/images/create returned error: exec: \"xz\": executable file not found in $PATH"
time="2022-02-28T01:20:41.320195242Z" level=info msg="shim containerd-shim started" address=/containerd-shim/5759075674e435004a4daa154d4d622830024f65fc88f27f668380072e9cb4e4.sock debug=false pid=588
time="2022-02-28T01:20:42.106275598Z" level=info msg="shim reaped" id=33f65322b7fef822803f823edd4832994471a3fb96b01a134e4d55df3d42f88a
time="2022-02-28T01:20:42.445521731Z" level=error msg="33f65322b7fef822803f823edd4832994471a3fb96b01a134e4d55df3d42f88a cleanup: failed to delete container from containerd: no such container"
time="2022-02-28T01:20:42.445606722Z" level=error msg="Handler for POST /v1.25/containers/33f65322b7fef822803f823edd4832994471a3fb96b01a134e4d55df3d42f88a/start returned error: cgroups: cgroup mountpoint does not exist: unknown"
time="2022-02-28T01:20:48.704942066Z" level=info msg="shim containerd-shim started" address=/containerd-shim/2f332b46911b581793eb49be516ae7d88ffb66dcc35eb2bc91ba66d79dda4139.sock debug=false pid=817
time="2022-02-28T01:20:49.299950798Z" level=info msg="shim reaped" id=9004e4244d5101163b10f38c6f5982b66c58e870dd3006b5a7449503fc424534
time="2022-02-28T01:20:49.829250435Z" level=error msg="9004e4244d5101163b10f38c6f5982b66c58e870dd3006b5a7449503fc424534 cleanup: failed to delete container from containerd: no such container"
time="2022-02-28T01:20:49.829321986Z" level=error msg="Handler for POST /v1.25/containers/9004e4244d5101163b10f38c6f5982b66c58e870dd3006b5a7449503fc424534/start returned error: cgroups: cgroup mountpoint does not exist: unknown"
time="2022-02-28T01:20:56.057441646Z" level=info msg="shim containerd-shim started" address=/containerd-shim/8bb5cf11abecd3458433532cde6b448750be1132fc4790e21cb88e3fda551fe0.sock debug=false pid=1029
time="2022-02-28T01:20:56.619058647Z" level=info msg="shim reaped" id=dd1d9ea5e572f070c2f72e3e4dba907908cd9f3c0bf75991372cc542593390fb
time="2022-02-28T01:20:56.929066353Z" level=error msg="dd1d9ea5e572f070c2f72e3e4dba907908cd9f3c0bf75991372cc542593390fb cleanup: failed to delete container from containerd: no such container"
time="2022-02-28T01:20:56.929145357Z" level=error msg="Handler for POST /v1.25/containers/dd1d9ea5e572f070c2f72e3e4dba907908cd9f3c0bf75991372cc542593390fb/start returned error: cgroups: cgroup mountpoint does not exist: unknown"
time="2022-02-28T01:30:52.457780916Z" level=info msg="shim containerd-shim started" address=/containerd-shim/12a3d207e4f8ad82b0d43e2a8953f1e1bda9c23d2611cbe0c210a65f8852acd8.sock debug=false pid=1251
time="2022-02-28T01:30:53.925225856Z" level=info msg="shim reaped" id=33f2c95b8eacf22bba2f8480ed1e947afacb370bc909cb527b8dfc54c550b455
time="2022-02-28T01:30:54.246293849Z" level=error msg="33f2c95b8eacf22bba2f8480ed1e947afacb370bc909cb527b8dfc54c550b455 cleanup: failed to delete container from containerd: no such container"
time="2022-02-28T01:30:54.246387704Z" level=error msg="Handler for POST /v1.25/containers/33f2c95b8eacf22bba2f8480ed1e947afacb370bc909cb527b8dfc54c550b455/start returned error: cgroups: cgroup mountpoint does not exist: unknown"
time="2022-02-28T01:30:59.179262838Z" level=info msg="shim containerd-shim started" address=/containerd-shim/e3c0b990725d394dd9298339f5236f82fb8057dbd5006ef67ce12582b5dc3d85.sock debug=false pid=1465
time="2022-02-28T01:30:59.845653311Z" level=info msg="shim reaped" id=12ea94bba53edb145abc1ac5bcc4fc60e82bb13b8f889cbafd2911f5bfaa6b8f
time="2022-02-28T01:31:00.391769662Z" level=error msg="12ea94bba53edb145abc1ac5bcc4fc60e82bb13b8f889cbafd2911f5bfaa6b8f cleanup: failed to delete container from containerd: no such container"
time="2022-02-28T01:31:00.391838217Z" level=error msg="Handler for POST /v1.25/containers/12ea94bba53edb145abc1ac5bcc4fc60e82bb13b8f889cbafd2911f5bfaa6b8f/start returned error: cgroups: cgroup mountpoint does not exist: unknown"
time="2022-02-28T01:31:04.798290638Z" level=info msg="shim containerd-shim started" address=/containerd-shim/49e4c282f323a79ec0f123273118a81934cb0895365ac95410fbc0c15b68c76b.sock debug=false pid=1679
time="2022-02-28T01:31:05.376424965Z" level=info msg="shim reaped" id=63fd5d823567c8d4d6598f2790ba3be0ded3d81a633e387ee20640d5851cfb3a
time="2022-02-28T01:31:06.028669401Z" level=error msg="63fd5d823567c8d4d6598f2790ba3be0ded3d81a633e387ee20640d5851cfb3a cleanup: failed to delete container from containerd: no such container"
time="2022-02-28T01:31:06.028736218Z" level=error msg="Handler for POST /v1.25/containers/63fd5d823567c8d4d6598f2790ba3be0ded3d81a633e387ee20640d5851cfb3a/start returned error: cgroups: cgroup mountpoint does not exist: unknown"
Did I get anything wrong here? Thanks.
Hi @himekifee, thanks for giving Sysbox a shot. On a quick pass, I would suspect some issue with the host kernel regarding cgroups. What host distro / kernel are you using? (e.g., cat /etc/os-release and uname -a).
Hi, it is up to date ArchLinux installation with kernel Linux 5.16.11-arch1-1 #1 SMP PREEMPT Thu, 24 Feb 2022 02:18:20 +0000 x86_64 GNU/Linux
Thanks @himekifee. Sysbox is not officially supported on ArchLinux yet, but happy to help you figure out what's going on.
Inside the Sysbox container (i.e., where the GitLab runner agent is running in your scenario), how does the cgroup file hierarchy look (i.e., tree -L 1 /sys/fs/cgroup)?
I ask because it seems the entity emitting the error seems to think there is a problem with it: Error response from daemon: cgroups: cgroup mountpoint does not exist: unknown. It appears that the GitLab Docker Executor is invoking Docker, and the latter is reporting the error.
Also, what happens if you docker exec into that Sysbox container and launch a Docker container manually (e.g., docker run -it alpine)?
root@bc4e3e71a570:/# tree -L 1 /sys/fs/cgroup
/sys/fs/cgroup
|-- cgroup.controllers
|-- cgroup.events
|-- cgroup.freeze
|-- cgroup.kill
|-- cgroup.max.depth
|-- cgroup.max.descendants
|-- cgroup.procs
|-- cgroup.stat
|-- cgroup.subtree_control
|-- cgroup.threads
|-- cgroup.type
|-- cpu.idle
|-- cpu.max
|-- cpu.max.burst
|-- cpu.pressure
|-- cpu.stat
|-- cpu.uclamp.max
|-- cpu.uclamp.min
|-- cpu.weight
|-- cpu.weight.nice
|-- cpuset.cpus
|-- cpuset.cpus.effective
|-- cpuset.cpus.partition
|-- cpuset.mems
|-- cpuset.mems.effective
|-- hugetlb.2MB.current
|-- hugetlb.2MB.events
|-- hugetlb.2MB.events.local
|-- hugetlb.2MB.max
|-- hugetlb.2MB.rsvd.current
|-- hugetlb.2MB.rsvd.max
|-- init.scope
|-- io.bfq.weight
|-- io.latency
|-- io.low
|-- io.max
|-- io.pressure
|-- io.prio.class
|-- io.stat
|-- io.weight
|-- memory.current
|-- memory.events
|-- memory.events.local
|-- memory.high
|-- memory.low
|-- memory.max
|-- memory.min
|-- memory.numa_stat
|-- memory.oom.group
|-- memory.pressure
|-- memory.stat
|-- memory.swap.current
|-- memory.swap.events
|-- memory.swap.high
|-- memory.swap.max
|-- misc.current
|-- misc.events
|-- misc.max
|-- pids.current
|-- pids.events
|-- pids.max
|-- rdma.current
`-- rdma.max
1 directory, 62 files
exec in the nestybox/gitlab-runner-docker container
Thanks; so there are 2 versions of cgroups, v1 and the newer v2, and your host has v2. FYI, the v1 hierarchy looks like this:
$ tree -L 1 /sys/fs/cgroup
/sys/fs/cgroup
├── blkio
├── cpu -> cpu,cpuacct
├── cpuacct -> cpu,cpuacct
├── cpu,cpuacct
├── cpuset
├── devices
├── freezer
├── hugetlb
├── memory
├── net_cls -> net_cls,net_prio
├── net_cls,net_prio
├── net_prio -> net_cls,net_prio
├── perf_event
├── pids
├── rdma
├── systemd
└── unified
It's fine to have cgroup v2 on the host, but I suspect the version of Docker running inside the Sysbox container is a bit old and does not support it, so it's complaining with cgroup mountpoint does not exist: unknown.
What's the version of Docker inside the Sysbox container? (docker version)
root@bc4e3e71a570:/# docker version
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:52 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.13
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:01:20 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.7
GitCommit: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
root@bc4e3e71a570:/# docekr info
bash: docekr: command not found
root@bc4e3e71a570:/# docker info
Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 19.03.13
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 5.16.11-arch1-1
Operating System: Ubuntu 20.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.763GiB
Name: bc4e3e71a570
ID: 5LEB:NP5C:2XKJ:P4GF:OLVS:SIUE:6QDG:6TVR:RFX3:TUX6:N254:XCMG
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares suppor
Thanks; I see, it's Docker 19.03.13 which does not support cgroups v2 (we need Docker 20.10).
What container image are you using? The reference nestybox/gitlab-runner-docker image or some other one?
Ya, I somehow got a realy old nestybox/gitlab-runner-docker latest acf0d0cbf731 16 months ago 1.03GB, but I just try doing this recently. Forgot where did I get it.
Yes, looks like it needs an update. Try this new one I just uploaded please:
nestybox/gitlab-runner-docker:21.10
If that works, I'll tag it as the latest.
By the way, the Dockerfile is here in case you want to update it or customize it in the future.
Starting service docker:20.10.12-dind ...
Pulling docker image docker:20.10.12-dind ...
Using docker image sha256:1a42336ff683d7dadd320ea6fe9d93a5b101474346302d23f96c9b4546cb414d for docker:20.10.12-dind with digest docker@sha256:6f2ae4a5fd85ccf85cdd829057a34ace894d25d544e5e4d9f2e7109297fedf8d ...
ERROR: Preparation failed: Error response from daemon: Unknown runtime specified sysbox-runc (docker.go:392:0s)
I think this is inside sys container? Because I was using one layer off, so runner on host and it worked fine.
Something is wrong, because there is no sysbox-runc inside the system container itself (there is just the GitLab runner and Docker).
Did you follow the steps in section GitLab Runner & Docker in a System Container of the blog?
I think I did. sysbox-runc is on the list
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 8
Server Version: 20.10.12
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: sysbox-runc io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 39259a8f35919a0d02c9ecc2871ddd6ccf6a7c6e.m
runc version: v1.1.0-0-g067aaf85
init version: de40ad0
Security Options:
seccomp
Profile: default
cgroupns
Kernel Version: 5.16.11-arch1-1
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.763GiB
Name: gitlab-runner-general
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
This is on the host. Also if you like, I can provide a tmate shell.
Is that config from the inner Docker? (i.e., the Docker inside the system container)
I meant that sysbox-runc should NOT be in the config of the inner Docker (i.e., because sysbox-runc lives at host level, there is no sysbox-runc inside the system container).
Not really, the one above is on the host, with sysbox-runc.
root@a3464053f254:/# docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 2
Server Version: 20.10.12
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
userxattr: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc version: v1.0.2-0-g52b36a2
init version: de40ad0
Security Options:
seccomp
Profile: default
cgroupns
Kernel Version: 5.16.11-arch1-1
Operating System: Ubuntu 20.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.763GiB
Name: a3464053f254
ID: NFAT:FJD6:5Y6B:6RTS:JWQB:KEV2:JCXN:EWDX:OKXR:LAB2:FYCJ:V5CG
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
The one from runner(inside the docker) does not have sysbox-runc.
So in the GitLab runner config you showed initially, you have:
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "my-docker-runner"
url = "https://git.me.asdf/"
token = "TOKEN"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:20.10.12"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache", "/var/lib/docker"]
shm_size = 0
runtime = "sysbox-runc"
If the GitLab runner is running inside the Sysbox container, then there is no sysbox-runc in it's context, so you can't configure it with sybox-runc as above.
You need something like:
[[runners]]
name = "syscont-runner-docker"
url = "https://gitlab.com/"
token = REGISTRATION_TOKEN
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:19.03.12"
privileged = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
Cool @himekifee, glad we got to the bottom of it. I've retagged the nestybox/gitlab-runner-docker:21.10 -> nestybox/gitlab-runner-docker:latest.
Thanks again for trying Sysbox, and let us know of any more questions.