[Bug] Latest Noble DinD Docker not working

[wolfpackmars2]

Describe the bug Attempting to run or build any docker images with the recently updated Noble DinD image fails. I am using kasmweb/ubuntu-noble-dind:1.18.0-rolling-weekly. The dind image was working yesterday.

I changed to the kasmweb/ubuntu-noble-dind:1.18.0 and the docker tools in the image work again. Something in the latest image seems to have broken the Docker inside docker.

Steps to reproduce the behavior: This can be tested easily by launching a clean kasmweb/ubuntu-noble-dind:1.18.0-rolling-weekly workspace and trying to start any docker container. Docker will fail to run with an overlayfs error.

Workspaces Version Version 1.18

Workspaces Installation Method Single Server

Client Browser (please complete the following information):

• OS: Ubuntu 24.04
• Browser chrome
• Version 142

[wolfpackmars2]

Unable to post a screenshot, but here is the output from within the workspace terminal

default:~$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
17eec7bbc9d7: Pull complete 
Digest: sha256:f7931603f70e13dbd844253370742c4fc4202d290c80442b2e68706d8f33ce26
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: failed to mount /tmp/containerd-mount4147353431: mount source: "overlay", target: "/tmp/containerd-mount4147353431", fstype: overlay, flags: 0, data: "workdir=/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/2/work,upperdir=/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/2/fs,lowerdir=/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs,index=off", err: invalid argument

Run 'docker run --help' for more information
default:~$ 

The problem goes away if I use the older 1.18.0 DinD image

[teja-kasm]

Hi there, thanks for reporting this issue. We’re aware of the problem and are actively working on a fix. Internal Ticket Reference: KASM-7979

I will update here once the issue has been resolved.

[wolfpackmars2]

Any additional information on this? Is this related to the new Docker-ce version 29 or something else?

[teja-kasm]

Yes, this is related to the Docker 29 update. The fix is on the way. Until the fix is applied, you can continue to use the non-rolling image kasmweb/ubuntu-noble-dind:1.18.0

[wolfpackmars2]

Installing working docker 28 on the rolling images at session launch is a temporary work around until the rolling images are fixed.

apt install \
    docker-ce=5:28.5.2-1~ubuntu.24.04~noble \
    docker-ce-cli=5:28.5.2-1~ubuntu.24.04~noble \
    docker-compose-plugin=2.40.3-1~ubuntu.24.04~noble \
    docker-ce-rootless-extras=5:28.5.2-1~ubuntu.24.04~noble \
    docker-buildx-plugin=0.30.1-1~ubuntu.24.04~noble

apt-mark hold \
    docker-ce \
    docker-ce-cli \
    docker-compose-plugin \
    docker-ce-rootless-extras \
    docker-buildx-plugin

I have tested these package versions to work, as long as there is not a need for Docker version 29.