dockercloud-agent icon indicating copy to clipboard operation
dockercloud-agent copied to clipboard

unhandled go err in "agent.go" or "reg.go" Registering in Docker Cloud via PATCH infinately hangs system

Open chrisdlangton opened this issue 8 years ago • 1 comments

~$ service dockercloud-agent restart

the log;

2017/12/21 18:00:02 Running dockercloud-agent: version 1.1.0
2017/12/21 18:00:02 Create pid file(/var/run/dockercloud-agent.pid): <pid was here>
2017/12/21 18:00:02 Checking if config file exists
2017/12/21 18:00:02 Loading Configuration file
2017/12/21 18:00:02 Registering in Docker Cloud via PATCH: https://cloud.docker.com/api/agent/v1/node/<UUID was here>

Program infinitely hangs, which also hangs dockerd..

to reproduce;

  • Install ubuntu-17.10-server-amd64
  • Follow the steps to Bring your own node
  • Notice much later the install finished and check the agent status on a new session;

~$ service dockercloud-agent status returns active (running)

  • tail the log tail -f /var/log/dockercloud/agent.log and restart the agent service dockercloud-agent restart
  • should show Registering in Docker Cloud via PATCH
  • try to use Docker docker ps and notice it just hangs

I'm no expert at golang, but followign the log message from here https://github.com/docker/dockercloud-agent/blob/d7eb171fcdf9a45b3fb5647cf862c7fda3ab83f0/agent.go#L72

to this call; https://github.com/docker/dockercloud-agent/blob/d7eb171fcdf9a45b3fb5647cf862c7fda3ab83f0/agent/reg.go#L59

I don;t see the error from GetCertificate so i assume it passes ok, i don't see the error from json.Marshal so again i assume it passes ok, then register is returned; https://github.com/docker/dockercloud-agent/blob/master/agent/reg.go#L123

which takes me to some loop where sendRegRequest is called and there is some complex err handling here i think it's here there is a black hole and the program has no options to continue or log.

EDIT: also, after the hang occurs and you issue service dockercloud-agent stop and try to use docker commands it complains about /var/run/docker.sock No such file or directory however it is fine; ls -la /var/run/docker.sock returns srw-rw---- 1 root docker 0 Dec 21 19:22 /var/run/docker.sock

Doing a service docker restart fixes it, you can do whatever docker commands you like without any issues until you run service dockercloud-agent start when it hangs again, and if you run stop the docker cli complains about the .sock file as before.

chrisdlangton avatar Dec 21 '17 07:12 chrisdlangton

I don't think the agent support ubuntu.17.x

This agent is supposed to mange the life cycle of docker. When the agent starts, it runs docker daemon as a subprocess. However, in ubuntu 17.x, the installation script failed to remove any pre-installed docker service. docker service are still managed by systemd, and the agent cannot start docker as a subprocess. I think that is why it hangs forever.

I would suggest to remove the pre-installed docker service completely, before running the installation script.

tifayuki avatar Dec 21 '17 18:12 tifayuki