dnsname icon indicating copy to clipboard operation
dnsname copied to clipboard
verified publisher icon containers

IP addresses aren't removed from addnhosts when using podman-compose

Open moqmar opened this issue 4 years ago • 3 comments

This sounds similar to #47, but it happens with Podman 3.3.1 and a build from the master branch for me:

  • podman-compose up -d
  • podman-compose down
  • podman-compose up -d
  • podman-compose down
  • Now, /run/user/0/containers/cni/dnsname/podman/addnhosts looks like this: 
    10.88.0.3	exampleproject
10.88.0.4	exampleproject
  • If I keep my project running & try to resolve the container, it's based on pure luck if I hit the correct IP - then everything works correctly though.
  • Interestingly, /var/lib/cni/networks/podman is empty.
  • If I run a container manually with podman run --rm -it --name test alpine, everything works as expected.
  • ~~Workaround for everyone finding this issue: if I use podman-compose -t identity up -d instead, everything works as expected (with the container name instead of the project/pod name).~~ ~~It seems weird to me that this only fails without -t identity, is that intended behaviour?~~ (edit: seems like this either worked only once or I've been lucky)
Output from podman network inspect podman 
[
    {
        "cniVersion": "0.4.0",
        "name": "podman",
        "plugins": [
            {
                "bridge": "cni-podman0",
                "hairpinMode": true,
                "ipMasq": true,
                "ipam": {
                    "ranges": [
                        [
                            {
                                "gateway": "10.88.0.1",
                                "subnet": "10.88.0.0/16"
                            }
                        ]
                    ],
                    "routes": [
                        {
                            "dst": "0.0.0.0/0"
                        }
                    ],
                    "type": "host-local"
                },
                "isGateway": true,
                "type": "bridge"
            },
            {
                "capabilities": {
                    "portMappings": true
                },
                "type": "portmap"
            },
            {
                "type": "firewall"
            },
            {
                "type": "tuning"
            },
            {
                "capabilities": {
                    "aliases": true
                },
                "domainName": "lab.example.org",
                "type": "dnsname"
            }
        ]
    }
]

moqmar avatar Dec 09 '21 19:12 moqmar

I'm facing the same issue on Rocky Linux 8.5 with podman/podman-plugins installed via dnf and podman-compose installed via pip.

[root@yebis-p0 tmp]# podman-compose --version
['podman', '--version', '']
using podman version: 3.3.1
podman-composer version  1.0.3
podman --version
podman version 3.3.1
exit code: 0

'podman network reload -a' will remove most of the stale entries but the IP address assigned to the last deceased container remains. (In my case, the IP address under /var/lib/cni/networks/tmp_default/ remains even though the container has been stopped.)

Here is the compose file that I've used for reproduction.

[root@yebis-p0 tmp]# cat /tmp/docker-compose.a.yml
[root@yebis-p0 tmp]# cat docker-compose.yml
version: "3"
services:
  test:
    container_name: bash-test
    image: bash
    command: sleep infinity

and the log.

[root@yebis-p0 tmp]# podman-compose up -d
['podman', '--version', '']
using podman version: 3.3.1
** excluding:  set()
['podman', 'network', 'exists', 'tmp_default']
podman run --name=bash-test -d --label io.podman.compose.config-hash=123 --label io.podman.compose.project=tmp --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=tmp --label com.docker.compose.project.working_dir=/tmp --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=test --net tmp_default --network-alias test bash sleep infinity
4ba346c334720be731c6f8d160fc83badffe2adb702be2e439c3e73cd7f2a368
exit code: 0
[root@yebis-p0 tmp]# podman-compose stop
['podman', '--version', '']
using podman version: 3.3.1
podman stop -t 10 bash-test
bash-test
exit code: 0
[root@yebis-p0 tmp]# cat /run/user/0/containers/cni/dnsname/tmp_default/addnhosts

10.88.3.20      bash-test       test
[root@yebis-p0 tmp]# podman-compose up -d
['podman', '--version', '']
using podman version: 3.3.1
** excluding:  set()
['podman', 'network', 'exists', 'tmp_default']
podman run --name=bash-test -d --label io.podman.compose.config-hash=123 --label io.podman.compose.project=tmp --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=tmp --label com.docker.compose.project.working_dir=/tmp --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=test --net tmp_default --network-alias test bash sleep infinity
Error: error creating container storage: the container name "bash-test" is already in use by "4ba346c334720be731c6f8d160fc83badffe2adb702be2e439c3e73cd7f2a368". You have to remove that container to be able to reuse that name.: that name is already in use
exit code: 125
podman start bash-test
bash-test
exit code: 0
[root@yebis-p0 tmp]# podman-compose stop
['podman', '--version', '']
using podman version: 3.3.1
podman stop -t 10 bash-test
bash-test
exit code: 0
[root@yebis-p0 tmp]# cat /run/user/0/containers/cni/dnsname/tmp_default/addnhosts

10.88.3.20      bash-test       test
10.88.3.21      bash-test       test
[root@yebis-p0 tmp]#

shinji-s avatar Feb 20 '22 19:02 shinji-s

podman-compose -d is just calls podman -d - https://github.com/containers/podman-compose/blob/9d5b2559274819e3b47230da85d4d306807bb4bf/podman_compose.py#L2440-L2442 .

The issue is not specific to podman-compose. With podman 3.4.4 and dnsname 1.3.1 entries for detached containers don't get removed from addnhosts when the container exits.

# podman run -d --rm --name test-detached --network net3 docker.io/library/busybox:latest sleep 3
d4ade6e4f58a39bf93878d2dd06716c1f1e34a873f8e10738f4dce8f795c9284

# cat /run/user/0/containers/cni/dnsname/net3/addnhosts 
10.89.3.2	test-detached

# podman run -it --rm --name test-interactive --network net3 docker.io/library/busybox:latest sleep 3

# cat /run/user/0/containers/cni/dnsname/net3/addnhosts 
10.89.3.2	test-detached

djoreilly avatar Sep 01 '22 11:09 djoreilly

After hacking the code to see the logs, I see this error

stat /run/containers/cni/dnsname/net3: no such file or directory

It should be /run/user/0/containers/cni/dnsname/net3.

The shell that runs the podman command has

# echo $XDG_RUNTIME_DIR
/run/user/0

And there is this https://github.com/containers/dnsname/blob/664c7a61f51c60ce2315472f49a71615a522ab8f/plugins/meta/dnsname/config.go#L61-L67

So maybe after the sleep 3 process exits, and because podman has detached and exited, something else comes along (conmon?) to call CNI DEL, but does not have XDG_RUNTIME_DIR in its environment?

This works because /run/containers/cni/dnsname/net3/ is used both at the start and end:

XDG_RUNTIME_DIR= podman run -d --rm --name web1 --network net3 docker.io/library/busybox:latest sleep 3

djoreilly avatar Sep 02 '22 08:09 djoreilly