Stuck injecting binary w/ Rancher Desktop

[codetent]

What happened?
When starting a workspace using the docker provider, building and starting the devcontainer seems to work but it is then stuck at Inject binary after receiving ARM-false after the pong message. This only occurs when using Rancher Desktop with docker CLI. I had no issues when using Docker Desktop directly.

What did you expect to happen instead?
It is not stuck at injecting and it proceeds with the remaining steps.

How can we reproduce the bug? (as minimally and precisely as possible)

1. Install Rancher Desktop and select docker CLI backend
2. Start Rancher Desktop
3. Use any devcontainer.json (I tried it with several images) and create a workspace with the docker provider
4. Open workspace (with --debug enabled)

Local Environment:

• DevPod Version: 0.3.4
• Operating System: windows
• ARCH of the OS: AMD64

DevPod Provider:

• Local/remote provider: docker

Anything else we need to know? I checked the steps in the inject code/script and the binary is successfully downloaded to the host. I then tried to attach a shell to the started container and then killing the cat process started by the inject script which seems to unblock the opening process. Therefore, it seems that there is some issue with sending the binary using cat to the container when using Rancher Desktop.

[FabianKramm]

@codetent thanks for creating this issue! Mhh strange, we will need to try to reproduce this. Are you using containerd or docker rancher desktop?

[codetent]

@codetent thanks for creating this issue! Mhh strange, we will need to try to reproduce this. Are you using containerd or docker rancher desktop?

I used dockerd as the container engine together with the docker CLI

[skython]

I'm getting the exact same behaviour. Is there any progress on that issue? Our team is also using Rancher Desktop and we would love to try out DevPod in our environment.

We are also using the dockerd (moby) engine in Rancher Desktop:

[wolfymaster]

I think I may have just hit this same issue - stuck on "Inject binary". I also use Rancher Desktop w/ the dockerd engine. Has anyone made progress on this?

Local Environment:

• DevPod Version: 0.4.0
• Operating System: windows

DevPod Provider:

• Local/remote provider: docker

[rcastagno]

Any news on this issue?

[hungerschnee]

I get the same Problem with Rancher Desktop and dockered engine. Any workaround found when working with Rancher Desktop?

Edit: This Problem only occurs with Rancher Desktop. Docker Desktop works just fine

[pascalbreuninger]

@rcastagno @hungerschnee Could you provide more details about your environment please? I've just tried this with the latest DevPod version, Rancher Desktop v1.13.1 with the dockerd engine and https://github.com/loft-sh/devpod-example-go as the repository and it worked fine

Edit: I just realised the original ticket is on window, are you guys on windows as well?

[doeringp]

I have the problem with Rancher Desktop v1.13.1 on Windows 10 22H2 (Build 19045.4170)

[phma4]

Same here with Rancher Desktop v1.13.1 on Windows 11 23H2 (Build 22631.3347)

Project: https://github.com/microsoft/vscode-remote-try-php DevPod Version: 0.5.6-12 DevPod log:

[09:22:01] info Execute SSH server command: C:\Program Files\DevPod\devpod-cli.exe helper sh -c 'C:\Program Files\DevPod\devpod-cli.exe' agent workspace up --workspace-info 'H4sIAAAAAAAA/7SSXU/bPBTHv8qjc503+qTtkjtW2KgGa9W1kzZZQo590hic2PJxCgj1u08mvHQMtKtdWj7H/r/87uHGuGuyXODCqa3qoIRZydiG0BFjy0ZpZW1ywZ1XHWOJxJ01kjFhOo+3nhiTWPNee8aeHyLGdiSMxNhhazzG3t3FtrEHI8kVmQ6il8+hvAcloYQ3NyGC/tVtPOayriECq4TvHUIJjfeWyjQ1Frut47ZJtso3fcWJ0FMiTJuimB5NUBZZPsVpLicTXuSTEZd5xrMCJ1jwrJIil3VRTQteZ0Vd/I9VleFYiEoWVdoq4QyZ2qfvCbXO7JREFxx1vA3CpBHX6CACY70yHYWrk8Xsy+nqcnm8PgvHHdf9b6M9oVsOT0kovetxH8H86/FsPf8+X/+4XM8vThebddh9Y3S/j6DlolFdSHYfgZJ4ICgkNOiHfQRkejc0sFV+hdaQ8sbdHSQ65PiQ4N8C2EcgHPLgc61aJM9bCyWMslEeZ3k8ytfZtDyalqPsJ0SgOfkNBdmvJkZH5fhDmHgELUQzgBZ+4FvsfBCsjeAaSgimQ/bcN48AL53ZOt7+90npAOQJ7paB24HfWGiV4G1Ykeam04bLzer8bcPa1D6mJh02U4caOSGlT4vpLkvGySTm2jY8GUEEqrtC4T8rP3MosfOKa3rRiLcohk6GrjUSHZ6DrUcfzzCojjzXwWjNNT2UJnrypg2L4aDV4gWth65fdfwneR838/OT09U7AEVPY2eLb+9BFv1ziH8BAAD//wEAAP//L9X8J54EAAA=' --debug
[09:22:01] info Use C:\Users\***********\.devpod\agent\contexts\default\workspaces\vscode-remote-try-php as workspace dir
[09:22:01] debug Created logger
[09:22:01] debug Received ping from agent
[09:22:01] debug Credentials server started on port 15109...
[09:22:02] debug Incoming client connection at /
[09:22:02] debug Credentials server started...
[09:22:02] debug Workspace Folder already exists C:\Users\**********\.devpod\agent\contexts\default\workspaces\vscode-remote-try-php\content
[09:22:02] debug Workspace exists, skip downloading
[09:22:02] debug Using docker command 'docker'
[09:22:02] debug execute inject script
[09:22:03] debug Received line after pong: ARM-false
[09:22:03] debug Inject binary

Docker info:

Client:
 Version:    25.0.4-rd
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.0
    Path:     C:\Users\******\.docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.7
    Path:     C:\Users\******\.docker\cli-plugins\docker-compose.exe

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 4e1fe7492b9df85914c389d1f15a3ceedbb280ac
 runc version: 51d5e94601ceffbbd85688df1c928ecccbfa4685
 init version:
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.146.1-microsoft-standard-WSL2
 Operating System: Rancher Desktop WSL Distribution
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.44GiB
 Name: JSW-AIP-NB060
 ID: 1262c7e4-0dbc-4bcb-99ae-0dabeab131c7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

[phma4]

I believe I'm able to narrow down the issue to the inject.sh script: https://github.com/loft-sh/devpod/blob/bb2f25dfb81127b338d4efb061744b73462fa9e1/pkg/inject/inject.sh#L56

Either "pong" is not being correctly passed here: IFS='$\n' read -r DEVPOD_PING

Or EOF (CTRL+D) is not transmitted here so the script moves on. $sh_c "cat > $INSTALL_PATH.$$"

I was successfully able to run the script inside the container by passing the right input to stdin, but I wasn't able to identify which of these lines are responsible

I'm still on Windows 11, but I've since updated devpod to 0.5.13 and Rancher Desktop to 1.14.1.

[wolfymaster]

@phma4jsw is correct. The inject.sh script is hanging - looks like w/e is supposed to be writing out the binary to /usr/local/bin/devpod.xx is not doing it. If I change the script to PREFER_DOWNLOAD=true then the binary is downloaded and I don't run into issues injecting. However, this requires running the script manually in the container.

If someone was able to point me in the direction of how this is supposed to work at a high level, then I would be happy to make a fix - however I don't have time to go hunt and figure out how those values are injected into inject.sh and what is supposed to be piping output.

[MihaiValentinGeorgescu]

How does devpod know which script to inject? I put mine in the /tmp folder.