containerd
Supplemental GID missing for group with same name as user
Description
Originally, I observed this with a mongodb container. The image user is mongodb and there is a group mongodb, but it's not the primary group of the user (nogroup is). containerd omits the mongodb group from the supplemental GID list.
Steps to reproduce the issue
- Create a GCP Debian Bookworm VM.
- Install latest k3s (v1.32.5+k3s1, comes with containerd v2.0.5-k3s1.32[^1]).
- Apply a specially crafted reproducer (source: https://github.com/burgerdev/weird-images/tree/74b32e2/src/gid)
kubectl apply -f https://raw.githubusercontent.com/burgerdev/weird-images/74b32e268da916576f307f6f997ae289322f5aa6/src/gid/gid.yaml
[^1]: This was the simplest way for me to reproduce the bug. I know that it's a bit removed from upstream containerd main, but since I believe to have found the root cause (see last section), it may be sufficient as a demo.
Describe the results you received and expected
Observed
In the container logs, I see the UID, GID and supplemental groups:
Uid: 2 2 2 2
Gid: 1 1 1 1
Groups: 1
Expected
The process should also have the supplemental group 2 (name).
What version of containerd are you using?
v2.0.5-k3s1.32
Any other relevant information
MongoDB setup
Image that I used when I discovered the bug: quay.io/mongodb/mongodb-community-server@sha256:8b73733842da21b6bbb6df4d7b2449229bb3135d2ec8c6880314d88205772a11
The unusual group setup comes from here, afaict:
https://github.com/mongodb/mongo/blob/cd9cc7450f3f665771a10f5c15f05bd6991c4d69/debian/mongodb-org-server.postinst#L25-L27
containerd bug
I assume it's this snippet that triggers the bug
https://github.com/containerd/containerd/blob/bcd000f443ef363b96e6dd4d67b4f6babfdef100/pkg/oci/spec_opts.go#L851-L854
The code seems to assume that a group with the same name as the user is always the primary group of the user.
It might be possible to just delete this check, given that the primary ID is always added
https://github.com/containerd/containerd/blob/bcd000f443ef363b96e6dd4d67b4f6babfdef100/pkg/oci/spec_opts.go#L833
but only if it hasn't been added already
https://github.com/containerd/containerd/blob/bcd000f443ef363b96e6dd4d67b4f6babfdef100/pkg/oci/spec_opts.go#L127-L135
Show configuration if it is related to CRI plugin.
No response
Heh, I was actually looking at some of these earlier Today, in preparation of upstreaming some code that was forked in BuildKit and Moby (docker engine), but there's quite some complexity in those code-paths, and there's also multiple impleementations which seem to slightly differ (between the CRI code and non-CRI code, although for some one wraps the other); https://github.com/containerd/containerd/blob/0bf07cd5c6ec01fadbf6606c3b4af76d2359d207/internal/cri/server/container_create_linux.go#L39-L64
Out of interest, I was wondering what a docker run would currently do (but I still need to look if it's currently depending on those code-paths from containerd, or using its own implementation);
Checking /etc/passwd and /etc/group;
docker run -it --rm --user mongodb mongodb/mongodb-community-server sh -c 'cat /etc/passwd | grep mongodb'
mongodb:x:101:65534::/home/mongodb:/usr/sbin/nologin
docker run -it --rm --user mongodb mongodb/mongodb-community-server sh -c 'cat /etc/group | grep mongodb'
mongodb:x:101:mongodb
docker run -it --rm --user mongodb mongodb/mongodb-community-server sh -c 'cat /etc/group | grep 65534'
nogroup:x:65534:
default user (as defined in the image)
docker run -it --rm mongodb/mongodb-community-server id
uid=101(mongodb) gid=65534(nogroup) groups=65534(nogroup),101(mongodb)
explicitly setting the user to "mongodb" (not passing a group);
docker run -it --rm --user mongodb mongodb/mongodb-community-server id
uid=101(mongodb) gid=65534(nogroup) groups=65534(nogroup),101(mongodb)
And with your reproducer image;
docker build \
-t test \
-f 'https://raw.githubusercontent.com/burgerdev/weird-images/74b32e268da916576f307f6f997ae289322f5aa6/src/gid/Containerfile' \
'https://github.com/burgerdev/weird-images.git#74b32e268da916576f307f6f997ae289322f5aa6:src/gid'
docker run -it --rm test
Uid: 2 2 2 2
Gid: 1 1 1 1
Groups: 1 2
^C
docker run -it --rm --entrypoint="" test id
uid=2(name) gid=1(daemon) groups=1(daemon),2(name)
edit: also tried with docker build to see if it's correct;
echo -e 'FROM mongodb/mongodb-community-server\nRUN id' | docker build --progress=plain --no-cache -
# ...
#6 [2/2] RUN id
#6 0.325 uid=101(mongodb) gid=65534(nogroup) groups=65534(nogroup),101(mongodb)
#6 DONE 0.4s
echo -e 'FROM mongodb/mongodb-community-server\nUSER root\nRUN id\nUSER mongodb\nRUN id' | docker build --progress=plain --no-cache -
#5 [2/3] RUN id
#5 0.232 uid=0(root) gid=0(root) groups=0(root)
#5 DONE 0.3s
#6 [3/3] RUN id
#6 0.307 uid=101(mongodb) gid=65534(nogroup) groups=65534(nogroup),101(mongodb)
#6 DONE 0.4s
Thanks for investigating this, @thaJeztah. I also tested this with docker and can confirm your observation, but discounted it because I thought docker may be introducing some special treatment here. Now I too wonder which code path results in the correct additionalGIDs - I've only found the one with the bug and one that does not look at /etc/group.
Thanks! Yes, it was also a bit "note to self" to compare the behavior; I had some "half-completed" branches to try and unify more of the implementations; especially to avoid (sometimes subtle) differences in behavior.
Big disclaimer that I'm not a kubernetes user, so it's possible that some differences were by design (https://github.com/kubernetes/enhancements/pull/3620), but I'm also aware that correct behavior for (supplemental) groups were a bit under-defined (also see https://github.com/opencontainers/image-spec/pull/1011, https://github.com/opencontainers/runtime-spec/issues/1180 for example).