libnetwork icon indicating copy to clipboard operation
libnetwork copied to clipboard
verified publisher icon moby

panic in findLBEndpointSandbox

Open JettJones opened this issue 7 years ago • 2 comments

Found this panic in the journalctl logs while investigating other container restarts. (During restart some running containers were terminated, which was why I was investigating).

Nov 08 08:34:25 host-000009 dockerd[11306]: time="2018-11-08T08:34:25.766619649Z" level=error msg="fatal task error" error="task: non-zero exit (1)" module=node/agent/taskmanager node.id=uh2gr37arkq5hxc6aqge2lf9l service.id=i5zpbufandth734h0krljnf6g task.id=5oz8aynoku5n0poe2s6454p8v
Nov 08 08:34:35 host-000009 dockerd[11306]: panic: runtime error: invalid memory address or nil pointer dereference
Nov 08 08:34:35 host-000009 dockerd[11306]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x557e4cd7ce40]
Nov 08 08:34:35 host-000009 dockerd[11306]: goroutine 623 [running]:
Nov 08 08:34:35 host-000009 dockerd[11306]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*endpoint).Name(0x0, 0x0, 0x0)
Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/endpoint.go:286 +0x30
Nov 08 08:34:35 host-000009 dockerd[11306]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*network).findLBEndpointSandbox(0xc420cfa000, 0x0, 0xc42084e000, 0x30, 0x7f16ab6cf6c8)
Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_linux.go:72 +0x16b
Nov 08 08:34:35 host-000009 dockerd[11306]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*network).rmLBBackend(0xc420cfa000, 0xc42219cf80, 0x10, 0x10, 0xc421fe8dc0, 0x101)Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_linux.go:179 +0x5a
Nov 08 08:34:35 host-000009 dockerd[11306]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).rmServiceBinding(0xc420194100, 0xc421f24920, 0x20, 0xc421f24940, 0x19, 0xc420d64ba0, 0x19, 0xc4216f5780, 0x40, 0xc420c40200, ...)
Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_common.go:377 +0xc2a
Nov 08 08:34:35 host-000009 dockerd[11306]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).handleEpTableEvent(0xc420194100, 0x557e4ddec860, 0xc4231a2230)
Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:951 +0x1398
Nov 08 08:34:35 host-000009 dockerd[11306]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).(github.com/docker/docker/vendor/github.com/docker/libnetwork.handleEpTableEvent)-fm(0x557e4ddec860, 0xc4231a2230)
Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:320 +0x40
Nov 08 08:34:35 host-000009 dockerd[11306]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).handleTableEvents(0xc420194100, 0xc421c71fe0, 0xc4202d4830)
Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:818 +0x53
Nov 08 08:34:35 host-000009 dockerd[11306]: created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).agentInit
Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:320 +0x524
Nov 08 08:34:35 host-000009 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 08 08:34:35 host-000009 systemd[1]: docker.service: Unit entered failed state.
Nov 08 08:34:35 host-000009 systemd[1]: docker.service: Failed with result 'exit-code'.
Nov 08 08:34:35 host-000009 systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Nov 08 08:34:35 host-000009 systemd[1]: Stopped Docker Application Container Engine.
Nov 08 08:34:35 host-000009 systemd[1]: Starting Docker Application Container Engine...

Docker Version 18.06.1:

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:56 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false

From docker info:

Kernel Version: 4.15.0-1025-azure
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.67GiB

JettJones avatar Nov 08 '18 18:11 JettJones

Same error on Docker 19.03.4 in swarm mode while running docker container prune -f from a container. Not sure if related since i have it run hourly for months without issue.

Nov 20 04:00:00 x dockerd[945]: time="2019-11-20T04:00:00.184566329-05:00" level=info msg="initialized VXLAN UDP port to 4789 "
Nov 20 04:00:00 x dockerd[945]: time="2019-11-20T04:00:00.417617908-05:00" level=warning msg="failed to deactivate service binding for container cleanup_cleanup_container_x.1.3eotglsya5nq9g4n68s3lx49d" error="No such container: cleanup_cleanup_container_x.1.3dmebtchrahpjsvx8avd7wk6e" module=node/agent node.id=ldgaickm7phmyiwzwgpsm356l
Nov 20 04:00:01 x dockerd[945]: time="2019-11-20T04:00:01.964165138-05:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 20 04:00:02 x dockerd[945]: time="2019-11-20T04:00:02.265195753-05:00" level=warning msg="10e7d7ac2c7e1c49be939e713bac79325cce162b8d35e6c162ed39f6be3cd3a3 cleanup: failed to unmount IPC: umount /var/lib/docker/containers/10e7d7ac2c7e1c49be939e713bac79325cce162b8d35e6c162ed39f6be3cd3a3/mounts/shm, flags: 0x2: no such file or directory"
Nov 20 04:00:02 x dockerd[945]: time="2019-11-20T04:00:02.379045539-05:00" level=info msg="initialized VXLAN UDP port to 4789 "
Nov 20 04:00:03 x dockerd[945]: panic: runtime error: invalid memory address or nil pointer dereference
Nov 20 04:00:03 x dockerd[945]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x55bb4e0327d0]
Nov 20 04:00:03 x dockerd[945]: goroutine 384 [running]:
Nov 20 04:00:03 x dockerd[945]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*endpoint).Name(0x0, 0x0, 0x0)
Nov 20 04:00:03 x dockerd[945]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/endpoint.go:286 +0x30
Nov 20 04:00:03 x dockerd[945]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*network).findLBEndpointSandbox(0xc0030ac1c0, 0xc000efb080, 0xc00192b0a0, 0x55bb4d20ef64, 0x3d)
Nov 20 04:00:03 x dockerd[945]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_linux.go:72 +0x142
Nov 20 04:00:03 x dockerd[945]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*network).rmLBBackend(0xc0030ac1c0, 0xc006023e30, 0x10, 0x10, 0xc0060ff280, 0x101)
Nov 20 04:00:03 x dockerd[945]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_linux.go:181 +0x58
Nov 20 04:00:03 x dockerd[945]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).rmServiceBinding(0xc00058a300, 0xc00460c8c0, 0x1b, 0xc00460c8e0, 0x19, 0xc00460c840, 0x19, 0xc000efab40, 0x40, 0xc000efac80, ...)
Nov 20 04:00:03 x dockerd[945]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_common.go:377 +0xbf8
Nov 20 04:00:03 x dockerd[945]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).handleEpTableEvent(0xc00058a300, 0x55bb4f9cd580, 0xc003f9bef0)
Nov 20 04:00:03 x dockerd[945]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:954 +0x12e4
Nov 20 04:00:03 x dockerd[945]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).handleTableEvents(0xc00058a300, 0xc000d12e00, 0xc000c24d00)
Nov 20 04:00:03 x dockerd[945]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:821 +0x53
Nov 20 04:00:03 x dockerd[945]: created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).agentInit
Nov 20 04:00:03 x dockerd[945]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:320 +0x4e1
Nov 20 04:00:03 x systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

Client: Docker Engine - Community
 Version:           19.03.4
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        9013bf583a
 Built:             Fri Oct 18 15:54:09 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.4
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       9013bf583a
  Built:            Fri Oct 18 15:52:40 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

 Kernel Version: 4.15.0-66-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.789GiB

codestation avatar Nov 20 '19 13:11 codestation

Same error here. Docker in swarm mode while restarting a container.

Dec 27 13:00:06 frogger dockerd[3496]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x55bcff158e50]
Dec 27 13:00:06 frogger dockerd[3496]: goroutine 1483 [running]:
Dec 27 13:00:06 frogger dockerd[3496]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*endpoint).Name(0x0, 0x0, 0x0)
Dec 27 13:00:06 frogger dockerd[3496]: #011/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/endpoint.go:286 +0x30
Dec 27 13:00:06 frogger dockerd[3496]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*network).findLBEndpointSandbox(0xc422137a40, 0x0, 0xc4214e6000, 0x30, 0x7f2abd9b1f78)
Dec 27 13:00:06 frogger dockerd[3496]: #011/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_linux.go:72 +0x16b
Dec 27 13:00:06 frogger dockerd[3496]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*network).rmLBBackend(0xc422137a40, 0xc4271a0840, 0x10, 0x10, 0xc422598800, 0x101)
Dec 27 13:00:06 frogger dockerd[3496]: #011/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_linux.go:179 +0x5a
Dec 27 13:00:06 frogger dockerd[3496]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).rmServiceBinding(0xc4201b7300, 0xc4271a07e0, 0xd, 0xc422ea4520, 0x19, 0xc422ea44e0, 0x19, 0xc42a8bb7c0, 0x40, 0xc425e15630, ...)
Dec 27 13:00:06 frogger dockerd[3496]: #011/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_common.go:377 +0xc2a
Dec 27 13:00:06 frogger dockerd[3496]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).handleEpTableEvent(0xc4201b7300, 0x55bd001c8820, 0xc422e11220)
Dec 27 13:00:06 frogger dockerd[3496]: #011/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:951 +0x1398
Dec 27 13:00:06 frogger dockerd[3496]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).(github.com/docker/docker/vendor/github.com/docker/libnetwork.handleEpTableEvent)-fm(0x55bd001c8820, 0xc422e11220)
Dec 27 13:00:06 frogger dockerd[3496]: #011/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:320 +0x40
Dec 27 13:00:06 frogger dockerd[3496]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).handleTableEvents(0xc4201b7300, 0xc421c344c0, 0xc420f060e0)
Dec 27 13:00:06 frogger dockerd[3496]: #011/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:818 +0x53
Dec 27 13:00:06 frogger dockerd[3496]: created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).agentInit
Dec 27 13:00:06 frogger dockerd[3496]: #011/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:320 +0x524

# docker version
Client:
 Version:           18.06.2-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        6d37f41
 Built:             Sun Feb 10 03:48:06 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.2-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       6d37f41
  Built:            Sun Feb 10 03:46:30 2019
  OS/Arch:          linux/amd64
  Experimental:     true

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:        16.04
Codename:       xenial
Kernel: 4.4.0-128-generic

luqpy avatar Dec 27 '19 17:12 luqpy