docker input not able to read podman socket

[tiny-pangolin]

Relevant telegraf.conf

[[inputs.docker]]
  endpoint = "unix:///var/run/podman/podman.sock"

Logs from Telegraf

2022-08-03T14:49:37Z I! Starting Telegraf 1.23.3
2022-08-03T14:49:37Z I! Loaded inputs: docker
2022-08-03T14:49:37Z I! Loaded aggregators: 
2022-08-03T14:49:37Z I! Loaded processors: 
2022-08-03T14:49:37Z W! Outputs are not used in testing mode!
2022-08-03T14:49:37Z I! Tags enabled: host=tomas.local.shiftsystems.net
2022-08-03T14:49:37Z W! Deprecated inputs: 0 and 2 options
2022-08-03T14:49:37Z D! [agent] Initializing plugins
2022-08-03T14:49:37Z D! [agent] Starting service inputs
2022-08-03T14:49:37Z E! [inputs.docker] Error in plugin: Cannot connect to the Docker daemon at unix:///var/run/podman/podman.sock. Is the docker daemon running?
2022-08-03T14:49:37Z E! [inputs.docker] Error in plugin: Cannot connect to the Docker daemon at unix:///var/run/podman/podman.sock. Is the docker daemon running?
2022-08-03T14:49:37Z D! [agent] Stopping service inputs
2022-08-03T14:49:37Z D! [agent] Input channel closed
2022-08-03T14:49:37Z D! [agent] Stopped Successfully
2022-08-03T14:49:37Z E! [telegraf] Error running agent: input plugins recorded 2 errors

System info

Telegraf 1.23.3 (git: HEAD 73121bd1), Fedora 36, Podman 4.1

Docker

No response

Steps to reproduce

1. enabled podman socket
2. set socket permissions to 666 instead of 660
3. run telegraf --debug --test --config /etc/telegraf/telegraf.d/podman.sock ...

Expected behavior

docker metrics should show from the CLI

Actual behavior

as regular user: 2022-08-03T14:52:48Z W! DeprecationWarning: Option "perdevice" of plugin "inputs.docker" deprecated since version 1.18.0 and will be removed in 2.0.0: use 'perdevice_include' instead 2022-08-03T14:52:48Z I! Starting Telegraf 1.23.3 2022-08-03T14:52:48Z I! Loaded inputs: docker 2022-08-03T14:52:48Z I! Loaded aggregators: 2022-08-03T14:52:48Z I! Loaded processors: 2022-08-03T14:52:48Z W! Outputs are not used in testing mode! 2022-08-03T14:52:48Z I! Tags enabled: host=tomas.local.shiftsystems.net 2022-08-03T14:52:48Z W! Deprecated inputs: 0 and 1 options 2022-08-03T14:52:48Z D! [agent] Initializing plugins 2022-08-03T14:52:48Z D! [agent] Starting service inputs 2022-08-03T14:52:48Z E! [inputs.docker] Error in plugin: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/podman/podman.sock: Get "http://%2Fvar%2Frun%2Fpodman%2Fpodman.sock/v1.24/info": dial unix /var/run/podman/podman.sock: connect: permission denied 2022-08-03T14:52:48Z E! [inputs.docker] Error in plugin: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/podman/podman.sock: Get "http://%2Fvar%2Frun%2Fpodman%2Fpodman.sock/v1.24/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0": dial unix /var/run/podman/podman.sock: connect: permission denied 2022-08-03T14:52:48Z D! [agent] Stopping service inputs 2022-08-03T14:52:48Z D! [agent] Input channel closed 2022-08-03T14:52:48Z D! [agent] Stopped Successfully 2022-08-03T14:52:48Z E! [telegraf] Error running agent: input plugins recorded 2 errors

as root:

2022-08-03T14:54:15Z W! DeprecationWarning: Option "perdevice" of plugin "inputs.docker" deprecated since version 1.18.0 and will be removed in 2.0.0: use 'perdevice_include' instead 2022-08-03T14:54:15Z I! Starting Telegraf 1.23.3 2022-08-03T14:54:15Z I! Loaded inputs: docker 2022-08-03T14:54:15Z I! Loaded aggregators: 2022-08-03T14:54:15Z I! Loaded processors: 2022-08-03T14:54:15Z W! Outputs are not used in testing mode! 2022-08-03T14:54:15Z I! Tags enabled: host=tomas.local.shiftsystems.net 2022-08-03T14:54:15Z W! Deprecated inputs: 0 and 1 options 2022-08-03T14:54:15Z D! [agent] Initializing plugins 2022-08-03T14:54:15Z D! [agent] Starting service inputs 2022-08-03T14:54:15Z E! [inputs.docker] Error in plugin: Cannot connect to the Docker daemon at unix:///var/run/podman/podman.sock. Is the docker daemon running? 2022-08-03T14:54:15Z E! [inputs.docker] Error in plugin: Cannot connect to the Docker daemon at unix:///var/run/podman/podman.sock. Is the docker daemon running? 2022-08-03T14:54:15Z D! [agent] Stopping service inputs 2022-08-03T14:54:15Z D! [agent] Input channel closed 2022-08-03T14:54:15Z D! [agent] Stopped Successfully 2022-08-03T14:54:15Z E! [telegraf] Error running agent: input plugins recorded 2 errors

Additional info

SELinux is set to permissive

[powersj]

I have an Ubuntu 22.04 system which has podman 3.4.4, and that seems to work:

ubuntu@nuci5a:~/telegraf-1.23.3/usr/bin$ ls -l /var/run/podman/podman.sock 
srw-rw---- 1 root root 0 Aug  4 14:06 /var/run/podman/podman.sock
ubuntu@nuci5a:~/telegraf-1.23.3/usr/bin$ sudo ./telegraf --config config.toml --test
2022-08-04T14:10:34Z W! DeprecationWarning: Option "perdevice" of plugin "inputs.docker" deprecated since version 1.18.0 and will be removed in 2.0.0: use 'perdevice_include' instead
2022-08-04T14:10:34Z I! Starting Telegraf 1.23.3
2022-08-04T14:10:34Z I! Loaded inputs: docker
2022-08-04T14:10:34Z I! Loaded aggregators: 
2022-08-04T14:10:34Z I! Loaded processors: 
2022-08-04T14:10:34Z W! Outputs are not used in testing mode!
2022-08-04T14:10:34Z I! Tags enabled: host=nuci5a
2022-08-04T14:10:34Z W! Deprecated inputs: 0 and 1 options
> docker,engine_host=nuci5a,host=nuci5a,server_version=3.4.4 n_containers=0i,n_containers_paused=0i,n_containers_running=0i,n_containers_stopped=0i,n_cpus=4i,n_goroutines=11i,n_images=0i,n_listener_events=0i,n_used_file_descriptors=12i 1659622234000000000
> docker,engine_host=nuci5a,host=nuci5a,server_version=3.4.4 memory_total=8200212480i 1659622234000000000

When I was able to get a version of 4.0 installed, I then started to see the same issue:

ubuntu@nuci5a:~/telegraf-1.23.3/usr/bin$ podman --version
podman version 4.0.3
ubuntu@nuci5a:~/telegraf-1.23.3/usr/bin$ ls -l /var/run/podman/podman.sock 
srw-rw---- 1 root root 0 Aug  4 14:06 /var/run/podman/podman.sock
ubuntu@nuci5a:~/telegraf-1.23.3/usr/bin$ sudo ./telegraf --config config.toml --test
2022-08-04T14:19:08Z W! DeprecationWarning: Option "perdevice" of plugin "inputs.docker" deprecated since version 1.18.0 and will be removed in 2.0.0: use 'perdevice_include' instead
2022-08-04T14:19:08Z I! Starting Telegraf 1.23.3
2022-08-04T14:19:08Z I! Loaded inputs: docker
2022-08-04T14:19:08Z I! Loaded aggregators: 
2022-08-04T14:19:08Z I! Loaded processors: 
2022-08-04T14:19:08Z W! Outputs are not used in testing mode!
2022-08-04T14:19:08Z I! Tags enabled: host=nuci5a
2022-08-04T14:19:08Z W! Deprecated inputs: 0 and 1 options
2022-08-04T14:19:08Z E! [inputs.docker] Error in plugin: Cannot connect to the Docker daemon at unix:///var/run/podman/podman.sock. Is the docker daemon running?
2022-08-04T14:19:08Z E! [inputs.docker] Error in plugin: Cannot connect to the Docker daemon at unix:///var/run/podman/podman.sock. Is the docker daemon running?
2022-08-04T14:19:08Z E! [telegraf] Error running agent: input plugins recorded 2 errors

It seems the client may need some sort of change in order to connect to the new version of podman. I did try updating the hard-coded version to 1.41 (latest) and that resulted in the same error.

[tiny-pangolin]

This issue is also occuring when trying to use the docker_log plugin

config:

[[inputs.docker_log]]
  endpoint = "unix:///var/run/podman.socket"

command:

telegraf --config /etc/telegraf/telegraf.d/podman.conf --test --debug

logs:

2022-08-06T13:09:15Z I! Starting Telegraf 1.23.3
2022-08-06T13:09:15Z I! Loaded inputs: docker_log
2022-08-06T13:09:15Z I! Loaded aggregators: 
2022-08-06T13:09:15Z I! Loaded processors: 
2022-08-06T13:09:15Z W! Outputs are not used in testing mode!
2022-08-06T13:09:15Z I! Tags enabled: host=tomas.local.shiftsystems.net
2022-08-06T13:09:15Z D! [agent] Initializing plugins
2022-08-06T13:09:15Z D! [agent] Starting service inputs
2022-08-06T13:09:15Z E! [inputs.docker_log] Error in plugin: Cannot connect to the Docker daemon at unix:///var/run/podman.socket. Is the docker daemon running?
2022-08-06T13:09:15Z D! [agent] Stopping service inputs
2022-08-06T13:09:15Z D! [agent] Input channel closed
2022-08-06T13:09:15Z D! [agent] Stopped Successfully
2022-08-06T13:09:15Z E! [telegraf] Error running agent: input plugins recorded 1 errors

[powersj]

@tiny6996 what does sudo podman info | grep sock report? For me it shows as /run/podman/podman.sock. I went to start looking at this again and re-installed podman (podman info) and now it works:

ubuntu@nuci5d:~/telegraf$ podman --version
podman version 4.0.1
ubuntu@nuci5d:~/telegraf$ sudo ls -l /run/podman/podman.sock
srw-rw---- 1 root root 0 Aug  8 16:24 /run/podman/podman.sock
$ cat config.toml 
[[outputs.file]]
[[inputs.docker]]
    endpoint = "unix:///run/podman/podman.sock"

ubuntu@nuci5d:~/telegraf$ sudo ./telegraf --debug --test --config ./config.toml
2022-08-08T16:31:05Z W! DeprecationWarning: Option "perdevice" of plugin "inputs.docker" deprecated since version 1.18.0 and will be removed in 2.0.0: use 'perdevice_include' instead
2022-08-08T16:31:05Z I! Starting Telegraf 1.24.0-75e8640a
2022-08-08T16:31:05Z I! Loaded inputs: docker
2022-08-08T16:31:05Z I! Loaded aggregators: 
2022-08-08T16:31:05Z I! Loaded processors: 
2022-08-08T16:31:05Z W! Outputs are not used in testing mode!
2022-08-08T16:31:05Z I! Tags enabled: host=nuci5d
2022-08-08T16:31:05Z W! Deprecated inputs: 0 and 1 options
2022-08-08T16:31:05Z D! [agent] Initializing plugins
2022-08-08T16:31:05Z D! [agent] Starting service inputs
> docker,engine_host=nuci5d,host=nuci5d,server_version=4.0.1 n_containers=0i,n_containers_paused=0i,n_containers_running=0i,n_containers_stopped=0i,n_cpus=4i,n_goroutines=11i,n_images=0i,n_listener_events=0i,n_used_file_descriptors=12i 1659976266000000000
> docker,engine_host=nuci5d,host=nuci5d,server_version=4.0.1 memory_total=8200200192i 1659976266000000000
2022-08-08T16:31:06Z D! [agent] Stopping service inputs
2022-08-08T16:31:06Z D! [agent] Input channel closed
2022-08-08T16:31:06Z D! [agent] Stopped Successfully

[telegraf-tiger[bot]]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!