kamal icon indicating copy to clipboard operation
kamal copied to clipboard
verified publisher icon basecamp

Trouble initializing local Docker registry (using `remote builder` or not)

Open charnould opened this issue 6 months ago • 4 comments

Hi, I've been waiting for this feature; So THANKS a lot! However I'm struggling setting it up...

Here my deploy.yaml: (I using an Apple M1 and deploying on AMD).

minimum_version: 2.8.2

service: pierre #<%= ENV['SERVICE'] %>

servers:
  web:
    - MY.IP.XXX #<%= ENV['IP'] %>

# image: pierre # new

registry:
  server: localhost:5555

ssh:
  port: 22 #<%= ENV['SSH_PORT'] %>

volumes:
  - 'datastores:/app/datastores/'

accessories:
  ollama:
    service: ollama
    image: ollama/ollama
    host: MY.IP.XXX #<%= ENV['IP'] %>
    options:
      gpus: all

proxy:
  ssl: true
  app_port: 3000
  # TODO: Make knowledge build a background job.
  # Otherwise request timeout if it takes too long.
  response_timeout: 600
  host: kamal.example.org # <%= ENV['HOST'] %>

builder:
  arch: amd64
  remote: ssh://[email protected] # ====> I try with and without this line !

env:
  clear:
    SERVICE: pierre # <%= ENV['SERVICE'] %>
  secret:
    - ANTHROPIC_API_KEY

Running Kamal setup (without remote builder) fails with the following message:

INFO [eb86c96c] Finished in 86.778 seconds with exit status 0 (successful).
Setting up local registry port forwarding to 46.224.2.167...
[email protected]'s password:
INFO [19535a12] Running docker image rm --force localhost:5555/pierre:d76ddb1c766475f22dea44434eafa6b4c900bdf6 on 46.224.2.167
INFO [19535a12] Finished in 0.293 seconds with exit status 0 (successful).
INFO [f939f210] Running docker pull localhost:5555/pierre:d76ddb1c766475f22dea44434eafa6b4c900bdf6 on 46.224.2.167
ERROR Error setting up port forwarding to 46.224.2.167: Errno::ERANGE: Result too large - send(2)
ERROR /opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/buffered_io.rb:101:in 'BasicSocket#send'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/buffered_io.rb:101:in 'Net::SSH::BufferedIo#send_pending'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:283:in 'block in Net::SSH::Connection::Session#ev_do_handle_events'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:282:in 'Array#each'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:282:in 'Net::SSH::Connection::Session#ev_do_handle_events'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/event_loop.rb:117:in 'Net::SSH::Connection::SingleSessionEventLoop#ev_select_and_postprocess'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/event_loop.rb:30:in 'Net::SSH::Connection::EventLoop#process'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:226:in 'Net::SSH::Connection::Session#process'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:179:in 'block in Net::SSH::Connection::Session#loop'
<internal:kernel>:168:in 'Kernel#loop'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:179:in 'Net::SSH::Connection::Session#loop'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:125:in 'Net::SSH::Connection::Session#close'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh.rb:268:in 'Net::SSH.start'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/kamal-2.8.2/lib/kamal/cli/build/port_forwarding.rb:33:in 'block (2 levels) in Kamal::Cli::Build::PortForwarding#forward_ports'
Finished all in 170.2 seconds
Releasing the deploy lock...
Finished all in 175.1 seconds
**ERROR (Errno::ERANGE): Result too large - send(2)**
error: "kamal" exited with code 1

Running Kamal setup (with builder: remote: ssh://[email protected]) fails with the following message:

INFO [c98e131b] Finished in 0.010 seconds with exit status 0 (successful).
Setting up local registry port forwarding to 46.224.2.167...
[email protected]'s password:
INFO [f65bb5a0] Running docker buildx inspect kamal-remote-ssh---root-46-224-2-167-local-registry | grep -q Endpoint:.*kamal-remote-ssh---root-46-224-2-167-local-registry-context && docker context inspect kamal-remote-ssh---root-46-224-2-167-local-registry-context --format '{{.Endpoints.docker.Host}}' | grep -xq ssh://[email protected] || (echo no compatible builder && exit 1) as charnould@localhost
**WARN Missing compatible builder, so creating a new one first**
INFO [1b1e2633] Running docker context rm kamal-remote-ssh---root-46-224-2-167-local-registry-context ; docker buildx rm kamal-remote-ssh---root-46-224-2-167-local-registry as charnould@localhost
INFO [fb4ee37c] Running /usr/bin/env true on 46.224.2.167
INFO [fb4ee37c] Finished in 0.140 seconds with exit status 0 (successful).
INFO [621b432f] Running docker context create kamal-remote-ssh---root-46-224-2-167-local-registry-context --description 'kamal-remote-ssh---root-46-224-2-167-local-registry host' --docker 'host=ssh://[email protected]' ; docker buildx create --name kamal-remote-ssh---root-46-224-2-167-local-registry --driver-opt network=host kamal-remote-ssh---root-46-224-2-167-local-registry-context as charnould@localhost
**ERROR Couldn't create remote builder: Permission denied, please try again.**
INFO [06106309] Running docker buildx build --output=type=registry --platform linux/amd64 --builder kamal-remote-ssh---root-46-224-2-167-local-registry -t localhost:5555/pierre:d76ddb1c766475f22dea44434eafa6b4c900bdf6 -t localhost:5555/pierre:latest --label service="pierre" --file Dockerfile . 2>&1 as charnould@localhost
 DEBUG [06106309] Command: ( export BUILDKIT_NO_CLIENT_TOKEN="1" ; docker buildx build --output=type=registry --platform linux/amd64 --builder kamal-remote-ssh---root-46-224-2-167-local-registry -t localhost:5555/pierre:d76ddb1c766475f22dea44434eafa6b4c900bdf6 -t localhost:5555/pierre:latest --label service="pierre" --file Dockerfile . 2>&1 )
 **DEBUG [06106309]       ERROR: no builder "kamal-remote-ssh---root-46-224-2-167-local-registry" found**
Finished all in 6.3 seconds
Releasing the deploy lock...
Finished all in 11.5 seconds
ERROR (SSHKit::Command::Failed): docker exit status: 256
docker stdout: ERROR: no builder "kamal-remote-ssh---root-46-224-2-167-local-registry" found
docker stderr: Nothing written

Any idea what's going on? Thanks a lot.

charnould avatar Oct 29 '25 09:10 charnould

I got the exact same issue with the exact same config for my own projects

Remotely I use Docker with this version: Docker version 28.3.1, build 38b7060

guillaumebriday avatar Nov 12 '25 04:11 guillaumebriday

I've made some progress but it still fails:

Setting up local registry port forwarding to XX.XXX.XXXX..
[email protected]'s password:
  INFO [e1e89ef5] Running docker image rm --force localhost:5555/charnould/hic:fc00027ee599fb89f7e199f51a64bd82716f4ca7 on XX.XXX.XXXX
  INFO [e1e89ef5] Finished in 0.269 seconds with exit status 0 (successful).
  INFO [89560b03] Running docker pull localhost:5555/charnould/hic:fc00027ee599fb89f7e199f51a64bd82716f4ca7 on XX.XXX.XXXX
 ERROR Error setting up port forwarding to XX.XXX.XXXX: Errno::ERANGE: Result too large - send(2)
 ERROR /opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/buffered_io.rb:101:in 'BasicSocket#send'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/buffered_io.rb:101:in 'Net::SSH::BufferedIo#send_pending'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:283:in 'block in Net::SSH::Connection::Session#ev_do_handle_events'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:282:in 'Array#each'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:282:in 'Net::SSH::Connection::Session#ev_do_handle_events'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/event_loop.rb:117:in 'Net::SSH::Connection::SingleSessionEventLoop#ev_select_and_postprocess'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/event_loop.rb:30:in 'Net::SSH::Connection::EventLoop#process'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:226:in 'Net::SSH::Connection::Session#process'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:179:in 'block in Net::SSH::Connection::Session#loop'
<internal:kernel>:168:in 'Kernel#loop'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:179:in 'Net::SSH::Connection::Session#loop'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:125:in 'Net::SSH::Connection::Session#close'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh.rb:268:in 'Net::SSH.start'
/opt/homebrew/lib/ruby/gems/3.4.0/gems/kamal-2.8.2/lib/kamal/cli/build/port_forwarding.rb:33:in 'block (2 levels) in Kamal::Cli::Build::PortForwarding#forward_ports'
  Finished all in 291.0 seconds
Releasing the deploy lock...
  Finished all in 295.2 seconds

Any clue?

charnould avatar Nov 12 '25 07:11 charnould

Perhaps, there is another process using this port on your server (your previous unfinished deployment, maybe?)

Go to your server and run this

lsof -i -P -n | grep LISTEN | grep 5555 # replace 5555 with the port you use for local registry

and kill the process by PID.

ATTENTION

Killing processes on the production server may lead to data loss or other issues. Make absolutely sure that the process you are killing is a previous instance of the local registry or another unnecessary program.

kalashnikovisme avatar Nov 12 '25 16:11 kalashnikovisme

@kalashnikovisme Thanks for your answer, but lsof -i -P -n | grep LISTEN | grep 5555 returns nothing, so nothing to kill...

I've applied all patches to my server, reboot it, delete all containers and volumes to start from a blank state... and it still does not work with the very same issue.

Any other idea?

charnould avatar Nov 13 '25 10:11 charnould

Hi guys,

Maybe it is a workaround for the moment (or not described in the Kamal documentation or not fully implemented yet in Kamal)

But it works with this deploy.yaml and the server: localhost:5555 value as a registry :

# Name of your application. Used to uniquely configure containers.
service: rails_dashboard

# Name of the container image.
image: louisbertin/rails_dashboard

# Deploy to these servers.
servers:
  web:
    - 167.x.x.x

# Enable SSL auto certification via Let's Encrypt and allow for multiple apps on a single web server.
# Remove this section when using multiple web servers and ensure you terminate SSL at your load balancer.
#
# Note: If using Cloudflare, set encryption mode in SSL/TLS setting to "Full" to enable CF-to-app encryption.
proxy:
  ssl: true
  hosts:
    - your-host.com

# Credentials for your image host.
registry:
  # Specify the registry server, if you're not using Docker Hub
  # server: registry.digitalocean.com / ghcr.io / ...
  #username: louisbertin

  # Always use an access token rather than real password when possible.
  #password:
  #  - KAMAL_REGISTRY_PASSWORD
  
  # local registry
  server: localhost:5555

# Inject ENV variables into containers (secrets come from .kamal/secrets).
env:
  secret:
    - RAILS_MASTER_KEY

# Aliases are triggered with "bin/kamal <alias>". You can overwrite arguments on invocation:
# "bin/kamal logs -r job" will tail logs from the first server in the job section.
aliases:
  console: app exec --interactive --reuse "bin/rails console"
  shell: app exec --interactive --reuse "bash"
  logs: app logs -f
  dbc: app exec --interactive --reuse "bin/rails dbconsole"


# Use a persistent storage volume for sqlite database files and local Active Storage files.
# Recommended to change this to a mounted volume path that is backed up off server.
volumes:
  - "rails_dashboard_storage:/rails/storage"

# Bridge fingerprinted assets, like JS and CSS, between versions to avoid
# hitting 404 on in-flight requests. Combines all files from new and old
# version inside the asset_path.
asset_path: /rails/public/assets

# Configure the image builder.
builder:
  # Build image via remote server (useful for faster amd64 builds on arm64 computers)
  remote: ssh://[email protected]:22
  arch: amd64

# Use a different ssh user than root
ssh:
  port: 22

At first, I encountered the same error mentionned before. But I needed to :

  1. Set the server: localhost:5555 value in the registry part
  2. Commit
  3. Run kamal registry setup ⚠️ important part

Then, after a docker ps command I can observe the following output on my local machine (MacBook Pro M1 - macOS 15.5) :

docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED         STATUS          PORTS                      NAMES
f96503c6ff66   registry:3                      "/entrypoint.sh /etc…"   8 minutes ago   Up 8 minutes    127.0.0.1:5555->5000/tcp   kamal-docker-registry
  1. Run kamal deploy and it works as I can see these lines (docker pull localhost:5555) in the command output logs :
DEBUG [eac50638] Finished in 0.571 seconds with exit status 0 (successful).
  INFO [3e6e4086] Running docker image rm --force localhost:5555/louisbertin/rails_dashboard:a6a4f439a32086fb279bfc80894c40d70886279b on 167.x.x.x
 DEBUG [3e6e4086] Command: docker image rm --force localhost:5555/louisbertin/rails_dashboard:a6a4f439a32086fb279bfc80894c40d70886279b
 DEBUG [3e6e4086]       Error response from daemon: No such image: localhost:5555/louisbertin/rails_dashboard:a6a4f439a32086fb279bfc80894c40d70886279b
  INFO [3e6e4086] Finished in 0.155 seconds with exit status 0 (successful).
  INFO [d8713a07] Running docker pull localhost:5555/louisbertin/rails_dashboard:a6a4f439a32086fb279bfc80894c40d70886279b on 167.x.x.x
 DEBUG [d8713a07] Command: docker pull localhost:5555/louisbertin/rails_dashboard:a6a4f439a32086fb279bfc80894c40d70886279b
 DEBUG [d8713a07]       a6a4f439a32086fb279bfc80894c40d70886279b: Pulling from louisbertin/rails_dashboard

ℹ️ i have only tested with the remote builder remote: ssh://[email protected]:22 but it should also work with the local one it is working even with the latest Docker version on my remote server : Docker version 29.0.1, build eedd969

LouisBertin avatar Nov 17 '25 15:11 LouisBertin

I faced the same issue:

...

 DEBUG [da88ffa7]       View build details: docker-desktop://dashboard/build/kamal-local-registry-docker-container/kamal-local-registry-docker-container0/ybvtplmw6svph4hh2y9bsrdor
  INFO [da88ffa7] Finished in 1.838 seconds with exit status 0 (successful).
Setting up local registry port forwarding to 82.x.x.x, myservice.com...
 ERROR Error setting up port forwarding to 82.x.x.x: RuntimeError: Failed to establish port forward on 82.x.x.x
 ERROR /Users/refacrot/.local/share/mise/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/kamal-2.8.2/lib/kamal/cli/build/port_forwarding.rb:36:in 'block (4 levels) in Kamal::Cli::Build::PortForwarding#forward_ports'
/Users/refacrot/.local/share/mise/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/service/forward.rb:226:in 'block in Net::SSH::Service::Forward#remote'
/Users/refacrot/.local/share/mise/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:605:in 'Net::SSH::Connection::Session#request_failure'
/Users/refacrot/.local/share/mise/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:549:in 'Net::SSH::Connection::Session#dispatch_incoming_packets'
/Users/refacrot/.local/share/mise/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:249:in 'Net::SSH::Connection::Session#ev_preprocess'
/Users/refacrot/.local/share/mise/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/event_loop.rb:104:in 'Array#each'
/Users/refacrot/.local/share/mise/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/event_loop.rb:104:in 'Net::SSH::Connection::SingleSessionEventLoop#ev_preprocess'
/Users/refacrot/.local/share/mise/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/event_loop.rb:28:in 'Net::SSH::Connection::EventLoop#process'
/Users/refacrot/.local/share/mise/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:226:in 'Net::SSH::Connection::Session#process'
/Users/refacrot/.local/share/mise/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:179:in 'block in Net::SSH::Connection::Session#loop'
<internal:kernel>:168:in 'Kernel#loop'
/Users/refacrot/.local/share/mise/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh/connection/session.rb:179:in 'Net::SSH::Connection::Session#loop'
/Users/refacrot/.local/share/mise/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/kamal-2.8.2/lib/kamal/cli/build/port_forwarding.rb:42:in 'block (3 levels) in Kamal::Cli::Build::PortForwarding#forward_ports'
/Users/refacrot/.local/share/mise/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/net-ssh-7.3.0/lib/net/ssh.rb:266:in 'Net::SSH.start'
/Users/refacrot/.local/share/mise/installs/ruby/3.4.7/lib/ruby/gems/3.4.0/gems/kamal-2.8.2/lib/kamal/cli/build/port_forwarding.rb:33:in 'block (2 levels) in Kamal::Cli::Build::PortForwarding#forward_ports'
  Finished all in 32.7 seconds
  ERROR (RuntimeError): Failed to establish port forward on 82.x.x.x

I have 2 nearly identical apps on the same VPS and the first one's deploy works just fine. The only difference between configs is that the second app has accessory of PostgreSQL and the first one doesn't have accessories at all.

On both apps I use local registry. Docker 29.0.1 Build eedd969

refacrot avatar Nov 20 '25 15:11 refacrot