[BUG] Unable to attach config file to read only filesystem

[s4s0l]

Description

I'm unable to attach config file to container when container has read-only filesystem.

Steps To Reproduce

networks:
  proxy-network:
    name: proxy-network

services:
  caddy:
    image: caddy:latest
    restart: unless-stopped
    container_name: caddy
    read_only: true
    networks:
      - proxy-network
    configs:
      - source: caddy_caddyfile
        target: /etc/caddy/Caddyfile

configs:
    caddy_caddyfile:
      content: |
        does not matter

docker compose up ends with:

[+] Running 1/2
 ✔ Network proxy-network  Created                                                                                                                                                                                                                                                                                0.1s 
 ⠋ Container caddy        Creating                                                                                                                                                                                                                                                                               0.0s 
Error response from daemon: container rootfs is marked read-only

Compose Version

v2.29.1

Docker Environment

Client: Docker Engine - Community
 Version:    27.1.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 27.1.1
 Storage Driver: overlay2
  Backing Filesystem: zfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-38-generic
 Operating System: Ubuntu 24.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.17GiB
 Name: my-nas
 ID: 4890d808-20cd-4a22-bed4-dde087d82fc2
 Docker Root Dir: /zfs/app-docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[glours]

Hello @s4s0l So it's a good catch, you identified a corner case error of the inline declaration 😅 What happened, when you use a config via a file declaration this one is mounted into the container even if the FS is in Read only mode. But it doesn't work the same with inline declaration, Compose write the content directly inside the container via a docker cp command which will obviously failed when the FS is in RO

If you're block by this situation, I suggest you to use a config file instead of using the inline declaration as I supposed you don't want to change your container FS mode

[s4s0l]

Thanks for the explanation, I ended up mounting config as volume (It didn't occur to me that config via file declaration could work any different).

I just want to point out that inlined configs is a great feature. I can interpolate some env variables in config files right in compose file. Doing so with bind mounts or config via a file declaration, usually requires some scripting around;)

[guillaumedsde]

But it doesn't work the same with inline declaration, Compose write the content directly inside the container via a docker cp command which will obviously failed when the FS is in RO

Hello @glours I tried using an inline compose config on a read only container but with a tmpfs mount like so:

services:
  test:
    image: busybox
    read_only: true
    tmpfs:
      - /config:mode=777
    configs:
      - source: test_config
        target: /config/test_config
    
configs:
  test_config:
    content: config_content

But running docker compose up (with docker 27.1.1 and compose 2.29.1) still yields:

Error response from daemon: container rootfs is marked read-only

My intuition was that this should have worked because the config file copy should happen after the tmpfs is mounted, do you know when the copy happens exactly?

(As @s4s0l I'm asking because the inline configuration is quite useful, notably because it interprets environment variables like the rest of the compose file)

[ambis]

Hit this myself just now. Thanks for the explanation, @glours

Saves me from trying to understand why this isn't working. I completely agree with @guillaumedsde as inlining configs with env support is super handy. Would be super nice to have this working.

[apollo13]

My intuition was that this should have worked because the config file copy should happen after the tmpfs is mounted, do you know when the copy happens exactly?

@guillaumedsde As far as I know it happens between container creation and starting the container. Last time I used wireshark to inspect the traffic between compose & docker the upload happens via https://docs.docker.com/reference/api/engine/version/v1.41/#tag/Container/operation/PutContainerArchive after the create. Since the host could be rebooted between create & start it doesn't really make sense (imo) to mount the bind mounts during create. So that most likely happens during start (probably defined in the OCI spec). This also means that your config file will not be visible since the tmpfs is mounted afterwards.

[ndeloof]

unfortunately there's no way with Docker engine API to prepare filesystem before container is started with custom content. I created a PR so we offer an explicit error message for this scenario.