for-mac exec invalid argument when running DinD containers since 4.33.0

Description

Hi

For testing and CI purposes we use DinD intensively, although in a slightly odd way. We install Docker inside a regular OS, such as Centos 7, Rockylinux 9 or Amazon 2023 and then execute Docker Compose base services on it. Real deployments run on proper servers, not containers, but this approach makes it cheaper and faster for CI purposes.

Until Docker for Mac 3.42.0 it has worked well but since 3.43.0 onward we get the following error when trying to execute any container on the nested Docker installation:

$ docker exec -it tmptest docker run -it alpine uname -a
exec /bin/uname: invalid argument

This is not happening on our CI pipelines, which are Linux based, but it happens on the Docker Desktop for Mac, which is hindering the local development.

Thanks!

Reproduce

Build an image with Docker

FROM rockylinux:9

RUN curl -o /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo && \
    yum -y install docker-ce docker-ce-cli containerd.io docker-compose-plugin && \
    systemctl enable docker
CMD ["/sbin/init"]

 docker build -t test -f Dockerfile .

Exec the container, which will be running Docker via systemctl, with the adequate privileges:

docker run -it --rm --name tmptest --privileged --mount type=tmpfs,destination=/tmp --mount type=tmpfs,destination=/run test

Execute a container on the nested Docker

docker exec -it tmptest docker run -it alpine uname -a

And we the following output

exec /bin/uname: invalid argument

Same happens with any container:

$ docker exec -it tmptest docker run -it traefik
exec /entrypoint.sh: invalid argument
$ docker exec -it tmptest docker run -it amazon/aws-cli
exec /usr/local/bin/aws: invalid argument

Expected behavior

Executing the same process using 4.32.0 and older versions is able to execute the command on the nested container without issues. Executing

docker exec -it tmptest docker run -it alpine uname -a

returns

Linux 488bebd0b14d 6.6.32-linuxkit #1 SMP Thu Jun 13 14:13:01 UTC 2024 aarch64 Linux

docker version

docker version
Client:
 Version:           27.2.0
 API version:       1.47
 Go version:        go1.21.13
 Git commit:        3ab4256
 Built:             Tue Aug 27 14:14:45 2024
 OS/Arch:           darwin/arm64
 Context:           default

Server: Docker Desktop 4.34.0 (165256)
 Engine:
  Version:          27.2.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       3ab5c7d
  Built:            Tue Aug 27 14:15:41 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.20
  GitCommit:        8fc6bcff51318944179630522a095cc9dbf9f353
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

docker info
Client:
 Version:    27.2.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2-desktop.1
    Path:     /Users/miquela/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2-desktop.2
    Path:     /Users/miquela/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     /Users/miquela/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     /Users/miquela/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/miquela/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /Users/miquela/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/miquela/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/miquela/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/miquela/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.13.0
    Path:     /Users/miquela/.docker/cli-plugins/docker-scout

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 45
 Server Version: 27.2.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 6.10.4-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 15.6GiB
 Name: docker-desktop
 ID: a006f68c-69b5-4905-9d0d-fce579f967b9
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/miquela/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

FDECBCC2-D397-427D-9F13-A1E398DEDE34/20240908205657

Additional Info

Running on a M1 MBP 2021

Sep 08 '24 21:09 madrover

While not ideal, I was able to get containers starting by configuring the storage driver to vfs.

Sep 10 '24 13:09 process

I can confirm that workaround works for me too, although due the big amount of space involved I might stick to version 4.32.0 until it's fixed. @process , might I ask how did you find that the issue was storage driver related?

Sep 10 '24 13:09 madrover

@madrover The short answer is "a hunch", but I know that's not a great reason 😁

When we initially set up DinD on Mac (around late 2021 I think), we had trouble starting containers. We changed the storage driver to fuse-overlayfs as a fix, so there was already a precedent of storage driver issues for us. I forget the exact details.
I thought maybe the exec [...]: invalid argument error was related to the Unix permissions, or some other filesystem metadata. This is probably wrong and unrelated, but I do think exec problems can often be caused by filesystem quirks.

Sep 11 '24 09:09 process

I just ran into this issue. On mac I am not able to set the storage driver to vfs (it results in an error). I'm on version 4.43.1.

root@docker-desktop:/var/asana-config/asana# docker run --rm --name some-redis redis
Unable to find image 'redis:latest' locally
latest: Pulling from library/redis
37259e733066: Pull complete
6724eb393acd: Pull complete
746d6e37cd6e: Pull complete
fcd9a200a986: Pull complete
e19c7e9db449: Pull complete
4f4fb700ef54: Pull complete
3b5fad749431: Pull complete
Digest: sha256:a1e0a3b3a6cedd74d8ba44805b2497b93851a296f08a44962fedc03f1c490b47
Status: Downloaded newer image for redis:latest
exec /usr/local/bin/docker-entrypoint.sh: invalid argument

When I change the format and try to run my docker (outer) image:

With vfs:

docker: Error response from daemon: InvalidArgument: snapshotter not loaded: vfs: invalid argument

With overlay2:

docker: Error response from daemon: InvalidArgument: snapshotter not loaded: overlay2: invalid argument

It's very unfortunate, since being able to run DinD is very powerful for local development!

Jul 11 '25 16:07 olafurkarl-asana