docker
Allow an environment variable to be used for an array field (e.g. devices:)
Not sure what the syntax would be exactly, but I would like to supply an optional and variable set of devices to an docker-compose.yml service.
Seems more flexible if the variable is a string and literally substituted, but maybe shell array variables should be used instead.
DEVICES= # empty set, as a string
DEVICES=() # or, empty set, as an array?
DEVICES='"/dev/sdb:/dev/sdb", "/dev/sdc:/dev/sdc"' # as a string
DEVICES=("/dev/sdb:/dev/sdb" "/dev/sdc:/dev/sdc") # or as an array?
foo:
devices: [ ${DEVICES} ] # not sure of the best syntax
image: ...
It would be great if this syntax allowed "appending" to an existing hard-coded array.
foo:
volumes: [ "/var/foo", ${EXTRA_VOLUMES} ] # not sure of the best syntax
image: ...
Without this feature, I can't use docker-compose directly and instead have to do unnatural things with yml substitution scripts and "docker-compose -f -".
PS: I don't know what's up with docker-compose and Swarm "services" on your roadmap. IMO Compose and the yml service definitions are a really nice way to define both simple and complicated environments.
Relevant Stackoverflow question: https://stackoverflow.com/questions/50919605/docker-compose-variable-substitution-interpolation-with-list-map-or-array-va
Please, implement this. Tuning networks list via orchestration of compose/swarm configs is really terrible.
how about the new version, is the new version resolved array environment variable issue ? @dustinlacewell Do you know if there have new solution about this?
In some cases, I had to use some flashy operations, just in certain situations.
extra_hosts: [
"${EXTRA_HOST_0-localhost0:127.0.0.1}", "${EXTRA_HOST_1-localhost1:127.0.0.1}",
"${EXTRA_HOST_2-localhost2:127.0.0.1}", "${EXTRA_HOST_3-localhost3:127.0.0.1}"
]
EXTRA_HOST_0=server0:10.0.0.10
EXTRA_HOST_1=server1:10.0.0.11
Any news about this issue? https://stackoverflow.com/questions/72459205/docker-compose-yaml-how-to-populate-single-line-array-from-environment-variable
I wonder we could support bash array syntax
devices: ${DEVICES[*]}
about the .env file format to define an array, using DEVICES =(xxx,yyy) could have some side effect on existing configuration, so I would suggest we require each value to be set:
DEVICES[0]="/dev/sda:/dev/sda"
DEVICES[1]="/dev/sdb:/dev/sdb"
DEVICES[2]="/dev/sdc:/dev/sdc"
we also could rely on bash append operator to distinguish arrays definition with use of parentheses in a regular value:
DEVICES+=("/dev/sda:/dev/sda" "/dev/sdb:/dev/sdb")
I just came up with a scenario for this: the Google SQL Auth Proxy requires a list of connection names to corresponding database servers. This could be passed as a single array environment variable and expand on command usage:
services:
cloud_sql_auth_proxy:
image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:latest
platform: linux/amd64
restart: unless-stopped
command: ["--unix-socket=/cloudsql", "${SQL_CONNECTIONS[*]}"]
user: root
@ndeloof If you're open to it, I could give this a stab, but since it's my first look at the codebases I might be vastly underestimating the issue here. So before I commit to anything, it would be cool to know if I'm at least close with the scope. So if we were to implement your suggestion with bash-like syntax (devices: ${DEVICES[*]}) in it's most basic form, we would need to:
- make changes to the spec (mainly interpolation I suppose?)
- implement the changes in compose-go
- write some kind of parsing for the new syntax (kinda like the template module)
- incorporate that parsing in the interpolation module
- add the test cases in relevant modules (wherever interpolation is used)
- bump the
compose-goversion & update some docs in the main repo
Am I even close? Or not at all?
Y'all need to simply comply with the YAML standard for sequences (single-line arrays). [ first, second, third ]
This will allow "first, second, third" to be passed and should just work and not interfere with anything.
As of June 17, 2025 this is still breaking.
If, e.g. networks: [ ${MYSQL_NETS:-db} ] is used with MYSQL_NETS=a,b,c,db (or anything) in .env, the error yaml: line [n]: did not find expected ',' or ']' is thrown. Why? It should always substitute ${} values before doing anything else, this is expected behavior...
If, on the other hand networks: ${MYSQL_NETS:-db} is used with MYSQL_NETS="[ a,b,c,db ]" in .env, the error validating /home/blah/blah/docker-compose.yml: services.db.networks must be a list is thrown. This is a bug: the value is a valid list.
This makes it impossible to specify multiple networks (for example, or any other sequence) when there are often constraints that prevent the use of any other format (e.g. for templates in portainer, or other deployment tools).
Sticking to standardized specs is great for many reasons, not the least of which being that it is what people familiar with the standard will expect. It's really not a great idea to deviate from spec by introducing a new syntax variation for something that has already been well defined.
I should note that in addition, csv arrays were accepted in previous versions of docker compose, so throwing errors now not only introduces a significant pain point (valid YAML is being rejected) but also breaks old code and is incredibly challenging to identify as problematic - especially when compose files are abstracted by deploy tools.
@Mugane you misunderstood how interpolation works in compose. This isn't just some yaml templating. Interpolation takes place after yaml has been parsed
I should note that in addition, csv arrays were accepted in previous versions of docker compose
Not sure what you mean, tested:
$ cat compose.yaml
services:
test:
image nginx
ports: [${PORT}]
$ cat .env
PORT=80,81
$ docker-compose-v1 config
ERROR: yaml.parser.ParserError: while parsing a flow sequence
in "./compose.yaml", line 4, column 11
expected ',' or ']', but got '{'
in "./compose.yaml", line 4, column 13
can you share a working example ?
I also wonder nobody reported this missing "existing feature" since we released Docker Compose v2 .. 4 years ago. Doesn't sound many noticed this was possible to use this interpolation hack in Compose v1
@ndeloof ugh this is convoluted. Why? Why then would specifying a .env file NOT load values from that .env file into the docker-compose.yml? Seems really confusing. Besides, you still need to provide a way to specify sequences. No matter how they are parsed, I see no reason to contradict the YAML spec. But if you insist, then who am I to question - please document/share the method (remember, single line by constraint)?
I've been trying to find the configuration where it worked, and I cannot - but there is one out there that did work, when I next boot into windows I'll find it, but that may take a while.
Why then would specifying a .env file NOT load values from that .env file into the docker-compose.yml ?
values are not loaded into the yaml but into the compose model. Yaml is parsed, then yaml tree leaves get interpolated based on environment. Again, interpolation is not a text file templating mechanism which takes place before yaml parsing.
@ndeloof I feel like you're hung up on explaining the internals of how the documented YAML format of the docker compose file format is handled and at at the same time missing the point - which is that there needs to be a way to specify sequences when there is a single-line constraint. Users are indicating that this is a missing capability that is affecting their workflow. I understand that there may be architectural decisions that complicate implementation, but surely that is just the thing that needs to be fixed, and not an excuse not to implement expected YAML syntax?