`failed to attach to the container: failed to open stdout fifo: error creating fifo binary:///usr/local/bin/nerdctl?_NERDCTL_INTERNAL_LOGGING=%2Fvar%2Flib%2Fnerdctl%2F1935db59: no such file or directory`

[AkihiroSuda]

Discussed in https://github.com/containerd/nerdctl/discussions/2876

^{Originally posted by haytok March 13, 2024}

Question

If we try to attach to a container that has been started as shown below, the following error will occur.

[haytok@lima-finch workspace]$ sudo nerdctl run --name test -d alpine sh -c "while true; do echo /'Hello, world/'; sleep 1; done"
54f605cda9b8184a5d16a543ba482dfaa1a90d3e71d9903c19b667bd202f5b18
[haytok@lima-finch workspace]$ sudo nerdctl ps
CONTAINER ID    IMAGE                              COMMAND                   CREATED          STATUS    PORTS    NAMES
54f605cda9b8    docker.io/library/alpine:latest    "sh -c while true; d…"    3 seconds ago    Up                 test
[haytok@lima-finch workspace]$ sudo nerdctl attach test
FATA[0000] failed to attach to the container: failed to open stdout fifo: error creating fifo binary:///usr/local/bin/nerdctl?_NERDCTL_INTERNAL_LOGGING=%2Fvar%2Flib%2Fnerdctl%2F1935db59: no such file or directory

Is this the expected behavior?

There seems to be an error in the following part of nerdctl, but I'm in trouble because I don't know how to fix it.

• https://github.com/containerd/nerdctl/blob/main/pkg/cmd/container/attach.go#L96C1-L99C3
• https://github.com/containerd/nerdctl/blob/9ce158bb9a259a999a33db54b43119ee36e50569/pkg/cioutil/container_io_unix.go#L122

Note that Docker doesn't cause such an error, and we can attach it to a container.

Environments

I'm using the following versions of containerd and nerdctl.

[haytok@lima-finch workspace]$ containerd --version
containerd github.com/containerd/containerd v1.7.13 7c3aca7a610df76212171d200ca3811ff6096eb8
[haytok@lima-finch workspace]$ nerdctl version
WARN[0000] failed to inspect the rootless containerd socket address  error="stat /run/user/504/containerd-rootless: no such file or directory"
Client:
 Version:	v1.7.3
 OS/Arch:	linux/amd64
 Git commit:	0a464409d0178e16d3d2bed36222937ec3fc9c77
 buildctl:
  Version:	v0.12.5
  GitCommit:	bac3f2b673f3f9d33e79046008e7a38e856b3dc6

Note that containerd and nerdctl are running in Lima launched from Finch.

[Shubhranshu153]

It seems

can't use log-uri with fifo-dir

https://github.com/containerd/containerd/blob/main/cmd/ctr/commands/tasks/exec.go

which makes the following case invalid

 else if flagD && logURI != "" {
		u, err := url.Parse(logURI)
		if err != nil {
		 	return nil, err
		 }
		 ioCreator = cio.LogURI(u)
	}

tried to

 else if flagD  {
		ioCreator = cio.NewCreator(cio.WithFIFODir("/run/containerd/fifo"))
	}

now it didnt throw errors but i didnt see any stdout. so not sure, but most likely i am not setting it up correctly.

[Shubhranshu153]

i think i was able to make it work

[root@lima-finch containerd]# sudo nerdctl attach test12
called in process terminal/Hello, world/
/Hello, world/
/Hello, world/
/Hello, world/
/Hello, world/
/Hello, world/
/Hello, world/
/Hello, world/
/Hello, world/
/Hello, world/

the change is

else if flagD  "" {
		ioCreator = cio.NewCreator(cio.WithStdio)
		fmt.Printf("in else loguri")

need to clean up these logic a bit and i dont exactly know if this is the correct ioCreator config, but seems to work as expected.

[mharwani]

Is this change compatible with nerdctl logs?

My understanding is that when a container is launched with -d flag, the stdout and stderr streams will be forwarded to the logging driver, so when you attach to this container there are no output fifos to open. Would probably need to have the output streams directed to both fifos and logging driver to make it work. There is a discussion on https://github.com/containerd/nerdctl/issues/1946 to support this dual logging feature, but looks like there hasn't been much progress.

[fahedouch]

Is this change compatible with nerdctl logs?

My understanding is that when a container is launched with -d flag, the stdout and stderr streams will be forwarded to the logging driver, so when you attach to this container there are no output fifos to open. Would probably need to have the output streams directed to both fifos and logging driver to make it work. There is a discussion on #1946 to support this dual logging feature, but looks like there hasn't been much progress.

This is implemeted here but not yet enabled for detach mode, I will take care of this

[Shubhranshu153]

that may not directly work the stdout binary seems to forward containers stdio to the logging binary, but in detached mode i presume containers stdio is closed? calling the modified function atleast didnt print anything with nerdctl logs but attach worked.

[mharwani]

Yeah, I'm wondering what the best approach is here. We need to keep a process alive to copy buffers from container fifos to a pipe used by logging. Or the logging driver can be restructured to access fifos directly.

[Shubhranshu153]

one approach can be to override the attach to pipe the logging. So when we attach we can read directly from the logdriver and print the output as you said.

didnt understand the logging driver can be restructured part.

[Shubhranshu153]

It seems we run /usr/local/bin/nerdctl _NERDCTL_INTERNAL_LOGGING /var/lib/nerdctl/1935db59 as part of nerdctl child process. In case when nerdctl cli launching container as fg via containerd, it has a wait for the container to exit for the main process to exit. As a result the logging child process doesnt exist. But in a detached mode the nerdctl cli will exit without waiting for the container to exit, as a result the logging cli would also exit.

If we pass in the logging uri containerd-shim handles the process (i am not completely sure but it appears so), as a result even if nerdctl cli exits the logging process keeps running.

The same dual logging approach we have for non-detached mode may not be viable here and we might need to come up with an alternate solution.

Let me know your opinion.

[fahedouch]

The same dual logging approach we have for non-detached mode may not be viable here and we might need to come up with an alternate solution.

agree. Apparently, the current architecture of containerd-shim does not allow it to be attached. I think we need to extend the responsibilities of the shim to fifo the stds and therefore when we want to attach, we will connect to this pipe.

[Shubhranshu153]

yes thats a good suggestion. Can i get assigned to this one, would like to implement it.