dockercloud-agent
dockercloud-agent copied to clipboard
docker-archive
Node registration to https://cloud.docker.com/ timed out (BYON behind firewall)
Hi,
We're planning to run dockercloud on some BYONs behind a firewall in our datacenter, so I'm doing some tests to see if it's going to work. I'm running vagrant/virtualbox on my laptop, ubuntu trusty 64 (14.04).
vagrant init ubuntu/trusty64
vagrant up
vagrant ssh
$ curl -Ls https://get.cloud.docker.com/ | sudo -H sh -s <big-long-key>
The agent took about 5 minutes, but eventually turned up in the Nodes tab in the Docker Cloud UI. All good. I shut the vagrant box down for the evening. Next morning I started the box up again, hoping that it would reconnect, it didn't, showing me the error Node registration to https://cloud.docker.com/ timed out in the logs.
What's weird is that the /var/log/dockercloud/ngrok.log shows that the periodic ping/pongs are working, giving the impression that a connection has been established.
I'm confused because it did work once, but has failed repeatedly since. I have verified that the box can curl https://cloud.docker.com/ successfully. I think it's pretty important to be able to restart a box and for it to reconnect.
The interesting part of /var/log/dockercloud/agent.log:
2016/09/07 07:08:04 Docker daemon (PID:1161) has been started
2016/09/07 07:08:06 Docker unix socket opened
2016/09/07 07:08:08 Node a7291e41-7537-4415-8e8d-0817d2b5e104.node.dockerapp.io is NOT publicly reachable
2016/09/07 07:08:08 Ngrok server: tunnel02.cloud.docker.com:4443
2016/09/07 07:08:08 Starting NAT tunnel
2016/09/07 07:08:09 Found new tunnel: tcp://tunnel02.cloud.docker.com:59835
2016/09/07 07:08:09 Sending tunnel address to Docker Cloud
2016/09/07 07:08:09 New tunnel has been set up
2016/09/07 07:13:04 Node registration to https://cloud.docker.com/ timed out
2016/09/07 07:13:04 Node state: Unreachable
The whole of /var/log/dockercloud/agent.log showing the successful initial connection:
2016/09/06 19:22:50 Running dockercloud-agent: version 1.1.0
2016/09/06 19:22:50 Create pid file(/var/run/dockercloud-agent.pid): 14826
2016/09/06 19:22:50 Checking if config file exists
2016/09/06 19:22:50 Loading Configuration file
2016/09/06 19:22:50 Registering in Docker Cloud via POST: https://cloud.docker.com/api/agent/v1/node/
2016/09/06 19:22:51 Cert CommonName has been changed from to a7291e41-7537-4415-8e8d-0817d2b5e104.node.dockerapp.io
2016/09/06 19:22:51 UUID has been changed from to a7291e41-7537-4415-8e8d-0817d2b5e104
2016/09/06 19:22:51 Updating configuration file...
2016/09/06 19:22:52 New TLS certificates generated
2016/09/06 19:22:52 Registering in Docker Cloud via PATCH: https://cloud.docker.com/api/agent/v1/node/a7291e41-7537-4415-8e8d-0817d2b5e104
2016/09/06 19:22:52 Downloading docker binary...
2016/09/06 19:22:52 Downloading docker definition from https://cloud.docker.com/api/tutum/v1/agent/docker/1.11.1-cs1/1.1.0.json
2016/09/06 19:22:52 Downloading docker from https://files.cloud.docker.com/packages/docker/docker-1.11.1-cs1.tgz
2016/09/06 19:23:18 Saving docker to /usr/bin/
2016/09/06 19:23:18 Uncompressing: /usr/bin/docker-containerd-ctr
2016/09/06 19:23:18 Uncompressing: /usr/bin/docker-runc
2016/09/06 19:23:18 Uncompressing: /usr/bin/docker
2016/09/06 19:23:19 Uncompressing: /usr/bin/docker-containerd
2016/09/06 19:23:19 Uncompressing: /usr/bin/docker-containerd-shim
2016/09/06 19:23:19 Found docker: version 1.11.1-cs1
2016/09/06 19:23:19 Initializing docker daemon
2016/09/06 19:23:19 Loading NAT tunnel module
2016/09/06 19:23:19 Verifying the registration with Docker Cloud
2016/09/06 19:23:19 Docker server started. Entering maintenance loop
2016/09/06 19:23:19 Starting docker daemon: [/usr/bin/docker daemon -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375 --userland-proxy=false --tlscert /etc/dockercloud/agent/cert.pem --tlskey /etc/dockercloud/agent/key.pem --tlscacert /etc/dockercloud/agent/ca.pem --tlsverify]
2016/09/06 19:23:19 Waiting for docker unix socket to be ready
2016/09/06 19:23:19 Docker daemon (PID:15680) has been started
2016/09/06 19:23:21 Docker unix socket opened
2016/09/06 19:23:23 Node a7291e41-7537-4415-8e8d-0817d2b5e104.node.dockerapp.io is NOT publicly reachable
2016/09/06 19:23:23 Downloading NAT tunnel binary ...
2016/09/06 19:23:23 Downloading ngrok definition from https://cloud.docker.com/api/tutum/v1/agent/ngrok/latest/1.1.0.json
2016/09/06 19:23:24 Downloading ngrok from https://files.cloud.docker.com/packages/ngrok/ngrok-1.7.tgz
2016/09/06 19:23:29 Saving ngrok to /usr/lib/dockercloud/
2016/09/06 19:23:29 Uncompressing: /usr/lib/dockercloud/._ngrok
2016/09/06 19:23:29 Uncompressing: /usr/lib/dockercloud/ngrok
2016/09/06 19:23:29 Ngrok server: tunnel02.cloud.docker.com:4443
2016/09/06 19:23:29 Starting NAT tunnel
2016/09/06 19:23:31 Found new tunnel: tcp://tunnel02.cloud.docker.com:41790
2016/09/06 19:23:31 Sending tunnel address to Docker Cloud
2016/09/06 19:23:31 New tunnel has been set up
2016/09/06 19:28:20 Node registration to https://cloud.docker.com/ succeeded
2016/09/06 19:30:02 Got signal: terminated
2016/09/06 19:30:02 Docker daemon is running
2016/09/06 19:30:02 Starting to shut down docker daemon gracefully
2016/09/06 19:30:02 Scheduling for shutting down, do not restart the tunnel
2016/09/07 07:08:02 Running dockercloud-agent: version 1.1.0
2016/09/07 07:08:02 Create pid file(/var/run/dockercloud-agent.pid): 854
2016/09/07 07:08:02 Checking if config file exists
2016/09/07 07:08:02 Loading Configuration file
2016/09/07 07:08:02 Registering in Docker Cloud via PATCH: https://cloud.docker.com/api/agent/v1/node/a7291e41-7537-4415-8e8d-0817d2b5e104
2016/09/07 07:08:04 Found docker: version 1.11.1-cs1
2016/09/07 07:08:04 Initializing docker daemon
2016/09/07 07:08:04 Loading NAT tunnel module
2016/09/07 07:08:04 Verifying the registration with Docker Cloud
2016/09/07 07:08:04 Docker server started. Entering maintenance loop
2016/09/07 07:08:04 Waiting for docker unix socket to be ready
2016/09/07 07:08:04 Starting docker daemon: [/usr/bin/docker daemon -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375 --userland-proxy=false --tlscert /etc/dockercloud/agent/cert.pem --tlskey /etc/dockercloud/agent/key.pem --tlscacert /etc/dockercloud/agent/ca.pem --tlsverify]
2016/09/07 07:08:04 Docker daemon (PID:1161) has been started
2016/09/07 07:08:06 Docker unix socket opened
2016/09/07 07:08:08 Node a7291e41-7537-4415-8e8d-0817d2b5e104.node.dockerapp.io is NOT publicly reachable
2016/09/07 07:08:08 Ngrok server: tunnel02.cloud.docker.com:4443
2016/09/07 07:08:08 Starting NAT tunnel
2016/09/07 07:08:09 Found new tunnel: tcp://tunnel02.cloud.docker.com:59835
2016/09/07 07:08:09 Sending tunnel address to Docker Cloud
2016/09/07 07:08:09 New tunnel has been set up
2016/09/07 07:13:04 Node registration to https://cloud.docker.com/ timed out
2016/09/07 07:13:04 Node state: Unreachable
I've done this again today. A new VM, new agent (new key).
I turn the VM off, wait for the Node status in the UI to be "UNREACHABLE". Then there's no reconnecting it.
