randomly exec: "plugin-git": executable file not found in $PATH

[healdropper]

Hi, I'm running a woodpecker server and agents on next version (master). I barely saw this error when I only had one worker but since I increased it to 16 workers, It happens way more often.

I'm using the default clone step in all my pipelines and I 've already read this documentaion https://woodpecker-ci.org/docs/next/usage/pipeline-syntax#clone

Thanks

[healdropper]

Using a custom clone config

clone: git: image: woodpeckerci/plugin-git

randomly gives this error

[qwerty287]

Which backend do you use?

[healdropper]

Backend? Not sure what do you mean, sorry

[qwerty287]

How are the commands executed? Docker, Kubernetes, SSH or locally?

[healdropper]

Docker, both server and agents configured in the same docker-compose file

[anbraten]

Which os and architecture is it?

[healdropper]

Hetzner Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Vendor ID: AuthenticAMD Model name: AMD Ryzen 9 5950X 16-Core Processor CPU family: 25 Model: 33 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1

[azmeuk]

Which backend do you use?

Same message exec: "plugin-git": executable file not found in $PATH with the latest git woodpecker running on debian. Also same message fork/exec woodpeckerci/plugin-git: no such file or directory when trying to override the clone configuration with the documentation example.

Strangely enough, everything worked fine before I rebooted the system. Now every job fails to clone.

Debian GNU/Linux 11 (bullseye) x86_64
Kernel: 5.10.0-21-amd64
go version go1.20.1 linux/amd64
Client: Docker Engine - Community
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        a5ee5b1
 Built:             Thu Feb  9 19:46:54 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       bc3805a
  Built:            Thu Feb  9 19:46:54 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.16
  GitCommit:        31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[azmeuk]

I could fix the issue by removing all the docker images and containers, then restarting the docker and woodpecker services. I am not sure what happened though.

docker rmi -f $(docker images -aq)
docker system prune -a --volumes
sudo systemctl restart docker

[6543]

hmm could be related to the default $PATH var used ... by docker

[6543]

but it should find it in any case: https://github.com/woodpecker-ci/plugin-git/blob/4447028fb10d4314930ed34c1dd1f178947b3a49/docker/Dockerfile.multiarch#L19

[6543]

https://stackoverflow.com/a/37904830 ; more

we might should try if moving to the "shell form" does fix it?

cc @anbraten I can create a test image that has only this change ... ... @healdropper would you try that one?

[healdropper]

Yes @6543 I could test it our CI server for sure, let me know

[healdropper]

@6543 any update about this? Thank you

[healdropper]

I created an internal image with the proposed change, I let you know @6543

[healdropper]

It definitely improved the ratio it happens, but still occurs 1 out of 15 executions.

[deltamualpha]

This has also just started happening for me, but 100% of the time. I've reverted a change I'd made that set WOODPECKER_MAX_WORKFLOWS to 4, but now every job fails with:

{"level":"warn","repo":"<snip>","pipeline":"77","workflow_id":"1251","error":"rpc error: code = Unknown desc = workflow finished with error check for git clone capabilities failed: exec: \"git\": executable file not found in $PATH","time":"2025-01-04T19:41:18Z","message":"cancel signal received"}

There's also a weird issue where the /tmp directory doesn't exist, either, unless I explicitly mount it.

[deltamualpha]

Oh, I figured out my problem, it's unrelated to this: I forgot to mount the docker socket in my agent container, so it was defaulting to the local backend... which doesn't really work out of the box inside the docker image.