for-linux
for-linux copied to clipboard
docker
Failed to update store for object type *libnetwork.sbState: timeout
- [x] This is a bug report
- [ ] This is a feature request
- [x] I searched existing issues before opening this one
Expected behavior
Container starts normally
Actual behavior
Container fails to start with message:
Handler for POST /containers/f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144/start returned error: failed to update the store state of sandbox: failed to update store for object type *libnetwork.sbState: timeout
Handler for POST /containers/631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:385: running prestart hook 0 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\\\\\\\\\"2018-05-03T12:57:32Z\\\\\\\\\\\\\\\" level=fatal msg=\\\\\\\\\\\\\\\"failed to update store for object type *libnetwork.sbState: timeout\\\\\\\\\\\\\\\"\\\\\\\\n\\\\\\\"\\\"\": unknown
Steps to reproduce the behavior
It seems to be happening randomly.
There are different workflows that are ran and they use different images with slightly different configuration (the only difference is the command they execute). However, the images and commands themselves should not be the problem, since they work 99% of the time. They all have the following in common:
- Create a container
- Start a container
- If starting the container failed, retry up to 5 times
Here are the full logs:
time="2018-05-03T12:19:11.280760099Z" level=info msg="libcontainerd: started new docker-containerd process" pid=3334
time="2018-05-03T12:19:12Z" level=info msg="starting containerd" module=containerd revision=9b55aab90508bd389d7654c4baf173a981477d55 version=v1.0.1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." module=containerd type=io.containerd.content.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." module=containerd type=io.containerd.snapshotter.v1
time="2018-05-03T12:19:12Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /mnt/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." module=containerd type=io.containerd.snapshotter.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." module=containerd type=io.containerd.metadata.v1
time="2018-05-03T12:19:12Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /mnt/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt"
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." module=containerd type=io.containerd.differ.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." module=containerd type=io.containerd.gc.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." module=containerd type=io.containerd.grpc.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." module=containerd type=io.containerd.grpc.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." module=containerd type=io.containerd.grpc.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." module=containerd type=io.containerd.grpc.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." module=containerd type=io.containerd.grpc.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." module=containerd type=io.containerd.grpc.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." module=containerd type=io.containerd.grpc.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." module=containerd type=io.containerd.grpc.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." module=containerd type=io.containerd.grpc.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." module=containerd type=io.containerd.monitor.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." module=containerd type=io.containerd.runtime.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." module=containerd type=io.containerd.grpc.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." module=containerd type=io.containerd.grpc.v1
time="2018-05-03T12:19:12Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." module=containerd type=io.containerd.grpc.v1
time="2018-05-03T12:19:12Z" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
time="2018-05-03T12:19:12Z" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
time="2018-05-03T12:19:12Z" level=info msg="containerd successfully booted in 0.089212s" module=containerd
time="2018-05-03T12:19:12.284828500Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2018-05-03T12:19:12.285045125Z" level=warning msg="Your kernel does not support swap memory limit"
time="2018-05-03T12:19:12.285101076Z" level=warning msg="Your kernel does not support cgroup rt period"
time="2018-05-03T12:19:12.285110903Z" level=warning msg="Your kernel does not support cgroup rt runtime"
time="2018-05-03T12:19:12.285704364Z" level=info msg="Loading containers: start."
time="2018-05-03T12:19:12.823183830Z" level=info msg="Loading containers: done."
[ 63.888085] aufs au_opts_verify:1597:dockerd[3331]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:19:13.330127079Z" level=info msg="Docker daemon" commit=7390fc6 graphdriver(s)=aufs version=17.12.1-ce
time="2018-05-03T12:19:13.331615457Z" level=info msg="Daemon has completed initialization"
time="2018-05-03T12:19:13.354929865Z" level=info msg="API listen on /var/run/docker.sock"
[ 143.048870] aufs au_opts_verify:1597:dockerd[4553]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 143.155169] aufs au_opts_verify:1597:dockerd[4553]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 143.457290] aufs au_opts_verify:1597:dockerd[4476]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:20:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/aba881de89494c2dd283fb8ed2abbaba0f640616dfaf4a709eb1b8c7fd41dc29/shim.sock" debug=false module="containerd/tasks" pid=9473
time="2018-05-03T12:20:33Z" level=info msg="shim reaped" id=aba881de89494c2dd283fb8ed2abbaba0f640616dfaf4a709eb1b8c7fd41dc29 module="containerd/tasks"
time="2018-05-03T12:20:33.867804679Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 147.235783] aufs au_opts_verify:1597:dockerd[3368]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 147.303928] aufs au_opts_verify:1597:dockerd[3368]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 147.397198] aufs au_opts_verify:1597:dockerd[4468]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:20:36Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3feb7c5b83daf36bf8e5d1e8622600a0155ce069ec93b773e7d6fb2d93743095/shim.sock" debug=false module="containerd/tasks" pid=10323
time="2018-05-03T12:20:36Z" level=info msg="shim reaped" id=3feb7c5b83daf36bf8e5d1e8622600a0155ce069ec93b773e7d6fb2d93743095 module="containerd/tasks"
time="2018-05-03T12:20:36.974804718Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 155.481175] aufs au_opts_verify:1597:dockerd[9475]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 155.549655] aufs au_opts_verify:1597:dockerd[9475]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 155.589105] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:20:44Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/e227edd391250ada7e68a97f3554366a979929c5ace3bbd0f34a1b15bd05af0d/shim.sock" debug=false module="containerd/tasks" pid=11131
[ 155.698247] aufs au_opts_verify:1597:dockerd[4502]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 155.742128] aufs au_opts_verify:1597:dockerd[4467]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 155.782631] aufs au_opts_verify:1597:dockerd[4502]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 155.782877] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 155.817885] aufs au_opts_verify:1597:dockerd[4467]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 155.818299] aufs au_opts_verify:1597:dockerd[9815]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 155.858138] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 155.890452] aufs au_opts_verify:1597:dockerd[9475]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 155.950695] aufs au_opts_verify:1597:dockerd[4532]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:20:45Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/f2f3aee1e75d7bf72fcb55e951cedb443602b36a94d1e6ee489032b6d67c6cb8/shim.sock" debug=false module="containerd/tasks" pid=11345
[ 156.163358] aufs au_opts_verify:1597:dockerd[3329]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 156.163886] aufs au_opts_verify:1597:dockerd[10464]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:20:45Z" level=info msg="shim reaped" id=e227edd391250ada7e68a97f3554366a979929c5ace3bbd0f34a1b15bd05af0d module="containerd/tasks"
[ 156.192095] aufs au_opts_verify:1597:dockerd[4440]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:20:45.139550485Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T12:20:45Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/92e9b02be5d4c801a8fa849eec57fd01610317fe2a61f43bfae6c31eec4dd3f2/shim.sock" debug=false module="containerd/tasks" pid=11534
time="2018-05-03T12:20:45Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/fb2809ffef46d84ee0e5cb1a7bc42c0d50f262892962a2aedd31bd6fcf5b43de/shim.sock" debug=false module="containerd/tasks" pid=11549
time="2018-05-03T12:20:45Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/cb239003a4e5705c2376366ee33d15535b2ed39995abace7a090c19b64f067b2/shim.sock" debug=false module="containerd/tasks" pid=11564
time="2018-05-03T12:20:45Z" level=info msg="shim reaped" id=cb239003a4e5705c2376366ee33d15535b2ed39995abace7a090c19b64f067b2 module="containerd/tasks"
time="2018-05-03T12:20:45.864568389Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 157.248851] aufs au_opts_verify:1597:dockerd[4465]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:20:46Z" level=info msg="shim reaped" id=92e9b02be5d4c801a8fa849eec57fd01610317fe2a61f43bfae6c31eec4dd3f2 module="containerd/tasks"
time="2018-05-03T12:20:46.242389901Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 157.303463] aufs au_opts_verify:1597:dockerd[4465]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:20:46Z" level=info msg="shim reaped" id=fb2809ffef46d84ee0e5cb1a7bc42c0d50f262892962a2aedd31bd6fcf5b43de module="containerd/tasks"
time="2018-05-03T12:20:46.293935744Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 157.406227] aufs au_opts_verify:1597:dockerd[3333]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:20:46Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/b752f08c0a5a0931326da147dbe9e9a53f34089265f5f787c675aa9a6c44b892/shim.sock" debug=false module="containerd/tasks" pid=12692
[ 158.051907] aufs au_opts_verify:1597:dockerd[9475]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 158.089208] aufs au_opts_verify:1597:dockerd[9475]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 158.157777] aufs au_opts_verify:1597:dockerd[3333]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 167.159891] aufs au_opts_verify:1597:dockerd[11548]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:20:58Z" level=info msg="shim reaped" id=f2f3aee1e75d7bf72fcb55e951cedb443602b36a94d1e6ee489032b6d67c6cb8 module="containerd/tasks"
time="2018-05-03T12:21:08.014300951Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 179.081653] aufs au_opts_verify:1597:dockerd[3333]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:21:21Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/6201fc70104cb91542fa7820b1de372c097855c15f3a9a2d958b039c5a696404/shim.sock" debug=false module="containerd/tasks" pid=16330
time="2018-05-03T12:22:27Z" level=info msg="shim reaped" id=b752f08c0a5a0931326da147dbe9e9a53f34089265f5f787c675aa9a6c44b892 module="containerd/tasks"
time="2018-05-03T12:22:27.164550966Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T12:22:57.637004871Z" level=warning msg="Failed to remove bridge endpoint dfff152 from store: timeout"
time="2018-05-03T12:24:01.548564064Z" level=warning msg="driver failed revoking external connectivity on endpoint frosty_edison (5ee3366e026c97d6aad592d43125798f2177fe6ae213bfe53704c42c69de9030): failed to update bridge endpoint 5ee3366 to store: failed to update bridge store for object type *bridge.bridgeEndpoint: timeout"
time="2018-05-03T12:32:27Z" level=info msg="shim reaped" id=6201fc70104cb91542fa7820b1de372c097855c15f3a9a2d958b039c5a696404 module="containerd/tasks"
time="2018-05-03T12:32:28.065619262Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 1212.977634] aufs au_opts_verify:1597:dockerd[4465]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1212.985499] aufs au_opts_verify:1597:dockerd[4507]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1213.168296] aufs au_opts_verify:1597:dockerd[4440]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1213.201052] aufs au_opts_verify:1597:dockerd[4561]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1222.718083] aufs au_opts_verify:1597:dockerd[4465]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1222.719501] aufs au_opts_verify:1597:dockerd[4561]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1222.724999] aufs au_opts_verify:1597:dockerd[4440]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1223.673502] aufs au_opts_verify:1597:dockerd[4440]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:38:45Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/8059579d4a3dac4662caffd622ba3447fb8ba0837bdba762ea15982a34b999a9/shim.sock" debug=false module="containerd/tasks" pid=19969
time="2018-05-03T12:39:38Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/ce2067dafad5a43e282ba997d26b5f3fa1a48b8f40e839caaeeb649a2da755c7/shim.sock" debug=false module="containerd/tasks" pid=20188
time="2018-05-03T12:40:33Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/576f305e407ca4a7b79fb98e7f7b32c8fb078129461479dbaef2b02806a9de28/shim.sock" debug=false module="containerd/tasks" pid=20392
[ 1378.754396] aufs au_opts_verify:1597:dockerd[3428]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1378.782306] aufs au_opts_verify:1597:dockerd[4556]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1378.791138] aufs au_opts_verify:1597:dockerd[4139]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:41:07Z" level=info msg="shim reaped" id=ce2067dafad5a43e282ba997d26b5f3fa1a48b8f40e839caaeeb649a2da755c7 module="containerd/tasks"
time="2018-05-03T12:41:07Z" level=info msg="shim reaped" id=8059579d4a3dac4662caffd622ba3447fb8ba0837bdba762ea15982a34b999a9 module="containerd/tasks"
time="2018-05-03T12:41:07.750518186Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T12:41:07.751598900Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 1378.851049] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1378.853413] aufs au_opts_verify:1597:dockerd[4468]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1378.905590] aufs au_opts_verify:1597:dockerd[4139]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1378.907240] aufs au_opts_verify:1597:dockerd[4468]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1378.918943] aufs au_opts_verify:1597:dockerd[4556]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:41:07Z" level=info msg="shim reaped" id=576f305e407ca4a7b79fb98e7f7b32c8fb078129461479dbaef2b02806a9de28 module="containerd/tasks"
time="2018-05-03T12:41:07.895838247Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 1379.409865] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:41:08Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/09297ad1266c38112fa91451c1aeba29f94a320a5e64eecb4ecde4cfefede43f/shim.sock" debug=false module="containerd/tasks" pid=20818
[ 1379.488907] aufs au_opts_verify:1597:dockerd[4217]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.489596] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.523135] aufs au_opts_verify:1597:dockerd[4468]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.553384] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.576458] aufs au_opts_verify:1597:dockerd[4139]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.576981] aufs au_opts_verify:1597:dockerd[10370]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:41:08Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/c328c0c5256256445a83c2208df79b983eb9c260e778ebc26a70a4c5fca32e3e/shim.sock" debug=false module="containerd/tasks" pid=20917
time="2018-05-03T12:41:08Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/69adbe0e43a0a68594c03276d764c80639090ed479208e63c3ff51a84fd40680/shim.sock" debug=false module="containerd/tasks" pid=20937
[ 1379.612506] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.636188] aufs au_opts_verify:1597:dockerd[4139]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.636657] aufs au_opts_verify:1597:dockerd[4450]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:41:08Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/f62eca4961c87162adf89653adba13c346ac23021903dbef6ef61fe9af1f6224/shim.sock" debug=false module="containerd/tasks" pid=21022
[ 1379.660105] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.660596] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.700206] aufs au_opts_verify:1597:dockerd[10370]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.700639] aufs au_opts_verify:1597:dockerd[4139]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.709506] aufs au_opts_verify:1597:dockerd[4483]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.768229] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.768714] aufs au_opts_verify:1597:dockerd[4450]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.769186] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:41:08Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/4e6af0c3a060ec819eb75cc18fb3a48aa96d6446aacb8e1db8c5cdf0ad318ff5/shim.sock" debug=false module="containerd/tasks" pid=21337
time="2018-05-03T12:41:08Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/2a37667e4e17106b503137d77fdacc90ac6bb4a4c1424f847cb6ec55248f0aa7/shim.sock" debug=false module="containerd/tasks" pid=21355
[ 1379.861948] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.880696] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.881208] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.908039] aufs au_opts_verify:1597:dockerd[4450]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.928463] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.929806] aufs au_opts_verify:1597:dockerd[4440]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.930317] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.959715] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.979409] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1379.979873] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.008926] aufs au_opts_verify:1597:dockerd[3333]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.012334] aufs au_opts_verify:1597:dockerd[4450]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:41:08Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/2409464b0e1d05204777991d6f38cd33e43baa27a4d94b60809d36411f344d15/shim.sock" debug=false module="containerd/tasks" pid=21527
[ 1380.059511] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.059752] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.124273] aufs au_opts_verify:1597:dockerd[3333]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.152168] aufs au_opts_verify:1597:dockerd[4451]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.153079] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.153500] aufs au_opts_verify:1597:dockerd[4451]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:41:09Z" level=info msg="shim reaped" id=09297ad1266c38112fa91451c1aeba29f94a320a5e64eecb4ecde4cfefede43f module="containerd/tasks"
time="2018-05-03T12:41:09Z" level=info msg="shim reaped" id=c328c0c5256256445a83c2208df79b983eb9c260e778ebc26a70a4c5fca32e3e module="containerd/tasks"
[ 1380.163374] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:41:09.081396504Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T12:41:09Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/cabe9ad6086046f00f2e8f87107f50a1013b25867f68ab16ac987cda32ee7e31/shim.sock" debug=false module="containerd/tasks" pid=21771
time="2018-05-03T12:41:09.089590325Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 1380.288256] aufs au_opts_verify:1597:dockerd[3329]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.298497] aufs au_opts_verify:1597:dockerd[9815]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:41:09Z" level=info msg="shim reaped" id=69adbe0e43a0a68594c03276d764c80639090ed479208e63c3ff51a84fd40680 module="containerd/tasks"
time="2018-05-03T12:41:09.243156513Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T12:41:09Z" level=info msg="shim reaped" id=f62eca4961c87162adf89653adba13c346ac23021903dbef6ef61fe9af1f6224 module="containerd/tasks"
time="2018-05-03T12:41:09.309550812Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 1380.867928] aufs au_opts_verify:1597:dockerd[9824]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1380.971829] aufs au_opts_verify:1597:dockerd[9824]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1381.156442] aufs au_opts_verify:1597:dockerd[4483]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1381.324949] aufs au_opts_verify:1597:dockerd[4466]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1381.380544] aufs au_opts_verify:1597:dockerd[4438]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1381.932019] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1382.299501] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1382.455690] aufs au_opts_verify:1597:dockerd[4438]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:41:11Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/863ed3cdd2b39dce772f5e318a8ae9cf7ce6768308ee2a86cfacdefd2d2bd6e3/shim.sock" debug=false module="containerd/tasks" pid=22330
[ 1382.488160] aufs au_opts_verify:1597:dockerd[3428]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1382.488878] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 1382.542161] aufs au_opts_verify:1597:dockerd[3348]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:41:11Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/0a845d16c6057edaa75a541923c69fe57c66ef2fe0657ca7a8689c721a97ee9e/shim.sock" debug=false module="containerd/tasks" pid=22433
[ 1382.570087] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:41:11Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/54c266c226c1eef82d481d82c5c0fb0fa6e8c1d96bff0f123a8a037e1995da41/shim.sock" debug=false module="containerd/tasks" pid=22471
time="2018-05-03T12:43:24Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910/shim.sock" debug=false module="containerd/tasks" pid=22762
time="2018-05-03T12:45:20Z" level=info msg="shim reaped" id=4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910 module="containerd/tasks"
time="2018-05-03T12:45:21.085680521Z" level=error msg="stream copy error: reading from a closed fifo"
time="2018-05-03T12:45:21.085684731Z" level=error msg="stream copy error: reading from a closed fifo"
time="2018-05-03T12:52:43.924774337Z" level=warning msg="Failed to remove bridge endpoint c61c5de from store: timeout"
time="2018-05-03T12:56:11Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e/shim.sock" debug=false module="containerd/tasks" pid=23228
time="2018-05-03T12:56:32.730379177Z" level=warning msg="Could not rollback container connection to network bridge"
time="2018-05-03T12:56:41.930879110Z" level=error msg="53fd32c54195dcb961a0c5eba182c0bd9088dd6c7043511e6f5b66153fec8469 cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T12:56:41.934737735Z" level=error msg="Handler for POST /containers/53fd32c54195dcb961a0c5eba182c0bd9088dd6c7043511e6f5b66153fec8469/start returned error: failed to create endpoint practical_fermi on network bridge: failed to save bridge endpoint d4455f1 to store: failed to update bridge store for object type *bridge.bridgeEndpoint: timeout"
time="2018-05-03T12:57:33Z" level=info msg="shim reaped" id=631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e module="containerd/tasks"
time="2018-05-03T12:57:33.594668367Z" level=error msg="stream copy error: reading from a closed fifo"
time="2018-05-03T12:57:33.594678064Z" level=error msg="stream copy error: reading from a closed fifo"
time="2018-05-03T12:58:29.298547311Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
[ 2420.401941] aufs au_opts_verify:1597:dockerd[10464]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T12:58:29.334262030Z" level=error msg="Handler for POST /containers/f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144/start returned error: failed to update the store state of sandbox: failed to update store for object type *libnetwork.sbState: timeout"
[ 2481.125782] aufs au_opts_verify:1597:dockerd[10370]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:04:09.562807189Z" level=error msg="455b514149e4e6d6f2ac9f83e18794790f35bae4a2be651eeaf0ad417d0c5565 cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T13:04:09.629206028Z" level=error msg="Handler for POST /containers/455b514149e4e6d6f2ac9f83e18794790f35bae4a2be651eeaf0ad417d0c5565/start returned error: failed to update store for object type *libnetwork.endpoint: timeout"
[ 2824.225697] aufs au_opts_verify:1597:dockerd[4438]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:22Z" level=info msg="shim reaped" id=4e6af0c3a060ec819eb75cc18fb3a48aa96d6446aacb8e1db8c5cdf0ad318ff5 module="containerd/tasks"
time="2018-05-03T13:05:22Z" level=info msg="shim reaped" id=863ed3cdd2b39dce772f5e318a8ae9cf7ce6768308ee2a86cfacdefd2d2bd6e3 module="containerd/tasks"
time="2018-05-03T13:05:22Z" level=info msg="shim reaped" id=cabe9ad6086046f00f2e8f87107f50a1013b25867f68ab16ac987cda32ee7e31 module="containerd/tasks"
time="2018-05-03T13:05:22Z" level=info msg="shim reaped" id=2a37667e4e17106b503137d77fdacc90ac6bb4a4c1424f847cb6ec55248f0aa7 module="containerd/tasks"
time="2018-05-03T13:05:22Z" level=info msg="shim reaped" id=2409464b0e1d05204777991d6f38cd33e43baa27a4d94b60809d36411f344d15 module="containerd/tasks"
time="2018-05-03T13:05:22Z" level=info msg="shim reaped" id=54c266c226c1eef82d481d82c5c0fb0fa6e8c1d96bff0f123a8a037e1995da41 module="containerd/tasks"
time="2018-05-03T13:05:22Z" level=info msg="shim reaped" id=0a845d16c6057edaa75a541923c69fe57c66ef2fe0657ca7a8689c721a97ee9e module="containerd/tasks"
time="2018-05-03T13:05:22.718232461Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:22.723032603Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:22.732704995Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:22.737503001Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:22.739781128Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:22.739992401Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:22.743739512Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:24.922085385Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T13:05:24.981522266Z" level=error msg="4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910 cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T13:05:24.986465355Z" level=error msg="Handler for POST /containers/4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:385: running prestart hook 0 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\\\\\\\\\"2018-05-03T12:45:20Z\\\\\\\\\\\\\\\" level=fatal msg=\\\\\\\\\\\\\\\"failed to update store for object type *libnetwork.sbState: timeout\\\\\\\\\\\\\\\"\\\\\\\\n\\\\\\\"\\\"\": unknown"
time="2018-05-03T13:05:25Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3fd7c0fa95e8a1740c7a7acee4303fe29ad3f2d13c8f617c3b041eb59f2909cc/shim.sock" debug=false module="containerd/tasks" pid=23860
time="2018-05-03T13:05:25Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/c0bae577c72fb15a6762d8a252fe11ba1457fa5c20e22643e838d43b846478da/shim.sock" debug=false module="containerd/tasks" pid=23861
time="2018-05-03T13:05:25.281302482Z" level=error msg="631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T13:05:25.281357469Z" level=error msg="Handler for POST /containers/631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:385: running prestart hook 0 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\\\\\\\\\"2018-05-03T12:57:32Z\\\\\\\\\\\\\\\" level=fatal msg=\\\\\\\\\\\\\\\"failed to update store for object type *libnetwork.sbState: timeout\\\\\\\\\\\\\\\"\\\\\\\\n\\\\\\\"\\\"\": unknown"
time="2018-05-03T13:05:25Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/60d4c30d5fa1be2bdcf552a523c90d17b9cff23eaa2ec0117d9752e27f64bba9/shim.sock" debug=false module="containerd/tasks" pid=23887
time="2018-05-03T13:05:26Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/5cb33d794fc97bda5d47d9b4c3cfac40ac845ad2e64512cb6cb7bab4f7fe420b/shim.sock" debug=false module="containerd/tasks" pid=24229
time="2018-05-03T13:05:26Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/53fd32c54195dcb961a0c5eba182c0bd9088dd6c7043511e6f5b66153fec8469/shim.sock" debug=false module="containerd/tasks" pid=24244
time="2018-05-03T13:05:26Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/455b514149e4e6d6f2ac9f83e18794790f35bae4a2be651eeaf0ad417d0c5565/shim.sock" debug=false module="containerd/tasks" pid=24252
time="2018-05-03T13:05:28Z" level=info msg="shim reaped" id=5cb33d794fc97bda5d47d9b4c3cfac40ac845ad2e64512cb6cb7bab4f7fe420b module="containerd/tasks"
time="2018-05-03T13:05:28Z" level=info msg="shim reaped" id=60d4c30d5fa1be2bdcf552a523c90d17b9cff23eaa2ec0117d9752e27f64bba9 module="containerd/tasks"
time="2018-05-03T13:05:28Z" level=info msg="shim reaped" id=3fd7c0fa95e8a1740c7a7acee4303fe29ad3f2d13c8f617c3b041eb59f2909cc module="containerd/tasks"
time="2018-05-03T13:05:28Z" level=info msg="shim reaped" id=c0bae577c72fb15a6762d8a252fe11ba1457fa5c20e22643e838d43b846478da module="containerd/tasks"
time="2018-05-03T13:05:28.143305737Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:28Z" level=info msg="shim reaped" id=455b514149e4e6d6f2ac9f83e18794790f35bae4a2be651eeaf0ad417d0c5565 module="containerd/tasks"
time="2018-05-03T13:05:28.145919096Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:28Z" level=info msg="shim reaped" id=53fd32c54195dcb961a0c5eba182c0bd9088dd6c7043511e6f5b66153fec8469 module="containerd/tasks"
time="2018-05-03T13:05:28.148071974Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:28.152767470Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:28.158309261Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:28.158373997Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 2839.943261] aufs au_opts_verify:1597:dockerd[9824]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2839.993879] aufs au_opts_verify:1597:dockerd[9824]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.041513] aufs au_opts_verify:1597:dockerd[9824]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:28Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/106ec50c4c12f2b948a4827d0ff0e1838295f491bc09887f36863d564909f94f/shim.sock" debug=false module="containerd/tasks" pid=25041
[ 2840.591499] aufs au_opts_verify:1597:dockerd[4217]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.624314] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.661426] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.662053] aufs au_opts_verify:1597:dockerd[4553]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.728855] aufs au_opts_verify:1597:dockerd[4556]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.729300] aufs au_opts_verify:1597:dockerd[3404]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.760177] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:29Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/d8083e074a477f8918abbb888b32b43c3b3bab844fb3dcf56a4b51d416fb4bc1/shim.sock" debug=false module="containerd/tasks" pid=25199
[ 2840.793708] aufs au_opts_verify:1597:dockerd[10199]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:29Z" level=info msg="shim reaped" id=106ec50c4c12f2b948a4827d0ff0e1838295f491bc09887f36863d564909f94f module="containerd/tasks"
[ 2840.824784] aufs au_opts_verify:1597:dockerd[10199]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:29.735931372Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:29Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/32a7db2ff75e9564522ffe66e87e9548b26be1abcd244c7e9ae43d68bd657fbd/shim.sock" debug=false module="containerd/tasks" pid=25310
[ 2840.837456] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.868224] aufs au_opts_verify:1597:dockerd[10199]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.916801] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2840.917439] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:29Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/1809be1799f5ac14d299eaaa7e72616d17e25bb81dbe9f56524228136b073680/shim.sock" debug=false module="containerd/tasks" pid=25404
[ 2841.083373] aufs au_opts_verify:1597:dockerd[3404]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.083952] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.101359] aufs au_opts_verify:1597:dockerd[4217]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.103338] aufs au_opts_verify:1597:dockerd[4467]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:30Z" level=info msg="shim reaped" id=d8083e074a477f8918abbb888b32b43c3b3bab844fb3dcf56a4b51d416fb4bc1 module="containerd/tasks"
[ 2841.165140] aufs au_opts_verify:1597:dockerd[4467]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.165367] aufs au_opts_verify:1597:dockerd[4467]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:30.086709122Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 2841.184674] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.206312] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.229293] aufs au_opts_verify:1597:dockerd[4556]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.243856] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2841.282163] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:30Z" level=info msg="shim reaped" id=32a7db2ff75e9564522ffe66e87e9548b26be1abcd244c7e9ae43d68bd657fbd module="containerd/tasks"
time="2018-05-03T13:05:30.216258406Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:30Z" level=info msg="shim reaped" id=1809be1799f5ac14d299eaaa7e72616d17e25bb81dbe9f56524228136b073680 module="containerd/tasks"
time="2018-05-03T13:05:30.390560846Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:30Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/f6b55fc3c592db3b5102dec234ff9a8ae12237e2c5ebece365dc809031423bbb/shim.sock" debug=false module="containerd/tasks" pid=25794
[ 2841.565448] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:30Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/88ad1363242aba92d1dacfb4a3105c4a41c410bff74c855c7ea78a518829a142/shim.sock" debug=false module="containerd/tasks" pid=25860
[ 2841.626850] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:30Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/6acbc487be0926aae2de82c8114da4c5c84c52f8aac078d2081ea8eabdb823a9/shim.sock" debug=false module="containerd/tasks" pid=25875
[ 2841.671400] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:30Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/95e37c5daee30c0a09be118877a7f5051345b55b737cbbe809e8a12aff2ecf60/shim.sock" debug=false module="containerd/tasks" pid=26043
time="2018-05-03T13:05:30Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/8d5c7b01b44ba99198dff801a03483ce9c3f450959272f2a8fb73583e74f85dd/shim.sock" debug=false module="containerd/tasks" pid=26174
time="2018-05-03T13:05:30Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/a073921268b279cf6e4f95cee7680c3a442f2aef9ddf0e410e1d4a453ac37216/shim.sock" debug=false module="containerd/tasks" pid=26179
time="2018-05-03T13:05:31Z" level=info msg="shim reaped" id=f6b55fc3c592db3b5102dec234ff9a8ae12237e2c5ebece365dc809031423bbb module="containerd/tasks"
time="2018-05-03T13:05:31.249550819Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:31Z" level=info msg="shim reaped" id=6acbc487be0926aae2de82c8114da4c5c84c52f8aac078d2081ea8eabdb823a9 module="containerd/tasks"
time="2018-05-03T13:05:31.277938818Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:31Z" level=info msg="shim reaped" id=88ad1363242aba92d1dacfb4a3105c4a41c410bff74c855c7ea78a518829a142 module="containerd/tasks"
time="2018-05-03T13:05:31.354977244Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:31Z" level=info msg="shim reaped" id=95e37c5daee30c0a09be118877a7f5051345b55b737cbbe809e8a12aff2ecf60 module="containerd/tasks"
time="2018-05-03T13:05:31.705973417Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 2842.858578] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:31Z" level=info msg="shim reaped" id=8d5c7b01b44ba99198dff801a03483ce9c3f450959272f2a8fb73583e74f85dd module="containerd/tasks"
time="2018-05-03T13:05:31.779888691Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:31Z" level=info msg="shim reaped" id=a073921268b279cf6e4f95cee7680c3a442f2aef9ddf0e410e1d4a453ac37216 module="containerd/tasks"
[ 2842.893561] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:31.809327571Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 2842.932558] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2842.953433] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.005402] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.005615] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.084807] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.157128] aufs au_opts_verify:1597:dockerd[4451]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.202115] aufs au_opts_verify:1597:dockerd[4451]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.277667] aufs au_opts_verify:1597:dockerd[4438]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.278244] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.278740] aufs au_opts_verify:1597:dockerd[4553]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.350171] aufs au_opts_verify:1597:dockerd[4553]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.350698] aufs au_opts_verify:1597:dockerd[4553]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.351271] aufs au_opts_verify:1597:dockerd[9076]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.401704] aufs au_opts_verify:1597:dockerd[4553]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/a150a284f2e4d7c13e5a01a677beed481c9476675b112242dc83cd15fd79e9fc/shim.sock" debug=false module="containerd/tasks" pid=26991
[ 2843.440154] aufs au_opts_verify:1597:dockerd[10548]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2843.450104] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3b0445e0f42c528c3f171cdcce12ef266ee1316439590a6ca404990afa55e20c/shim.sock" debug=false module="containerd/tasks" pid=27053
time="2018-05-03T13:05:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/61f470a5765d2bc3e00e2e63d8f5c05c4c7e495a97e8581d40fec26564104b5e/shim.sock" debug=false module="containerd/tasks" pid=27100
time="2018-05-03T13:05:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3704e7e21501a702889c37174e2776205d3cb4cba7a2e0e245b53d7b129c978f/shim.sock" debug=false module="containerd/tasks" pid=27143
time="2018-05-03T13:05:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/4ee08b443e2d4842de3f71e68cb3741a33f8d480b0ecb7f11f4d27c12c048f16/shim.sock" debug=false module="containerd/tasks" pid=27162
time="2018-05-03T13:05:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/7d7aa80cd7243185ceb02008d180cd9a05ac5424e38c40d1a4cbfb9531e9912d/shim.sock" debug=false module="containerd/tasks" pid=27167
time="2018-05-03T13:05:33Z" level=info msg="shim reaped" id=a150a284f2e4d7c13e5a01a677beed481c9476675b112242dc83cd15fd79e9fc module="containerd/tasks"
time="2018-05-03T13:05:33.131708056Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:33Z" level=info msg="shim reaped" id=3704e7e21501a702889c37174e2776205d3cb4cba7a2e0e245b53d7b129c978f module="containerd/tasks"
time="2018-05-03T13:05:33.210302701Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:33Z" level=info msg="shim reaped" id=61f470a5765d2bc3e00e2e63d8f5c05c4c7e495a97e8581d40fec26564104b5e module="containerd/tasks"
time="2018-05-03T13:05:33.288034359Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:33Z" level=info msg="shim reaped" id=7d7aa80cd7243185ceb02008d180cd9a05ac5424e38c40d1a4cbfb9531e9912d module="containerd/tasks"
time="2018-05-03T13:05:33Z" level=info msg="shim reaped" id=4ee08b443e2d4842de3f71e68cb3741a33f8d480b0ecb7f11f4d27c12c048f16 module="containerd/tasks"
time="2018-05-03T13:05:33Z" level=info msg="shim reaped" id=3b0445e0f42c528c3f171cdcce12ef266ee1316439590a6ca404990afa55e20c module="containerd/tasks"
time="2018-05-03T13:05:33.342540748Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:33.343263591Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:05:33.343715988Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 2844.722691] aufs au_opts_verify:1597:dockerd[4561]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2844.786017] aufs au_opts_verify:1597:dockerd[10370]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2844.828578] aufs au_opts_verify:1597:dockerd[4532]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:05:33Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/e9512f9a0e6305712e944bf79d2c18cc4dba2e86c0ab867b2d8431ad7b6e1910/shim.sock" debug=false module="containerd/tasks" pid=27949
time="2018-05-03T13:05:34Z" level=info msg="shim reaped" id=e9512f9a0e6305712e944bf79d2c18cc4dba2e86c0ab867b2d8431ad7b6e1910 module="containerd/tasks"
time="2018-05-03T13:05:34.609214165Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 2896.528035] aufs au_opts_verify:1597:dockerd[10199]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2896.846865] aufs au_opts_verify:1597:dockerd[4483]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:06:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e/shim.sock" debug=false module="containerd/tasks" pid=28345
time="2018-05-03T13:06:32Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910/shim.sock" debug=false module="containerd/tasks" pid=28346
[ 2955.720124] aufs au_opts_verify:1597:dockerd[3404]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:07:27.683705899Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
[ 3079.979644] aufs au_opts_verify:1597:dockerd[28486]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3082.613489] aufs au_opts_verify:1597:dockerd[28486]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3087.228776] aufs au_opts_verify:1597:dockerd[3404]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:09:45Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/24d7d21c2576e8ec2199461ef5d5ed8df3ea6d5e0a8c5a7defc56505532e215b/shim.sock" debug=false module="containerd/tasks" pid=28778
[ 3198.490768] aufs au_opts_verify:1597:dockerd[3346]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:11:27.894633572Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T13:18:12Z" level=info msg="shim reaped" id=24d7d21c2576e8ec2199461ef5d5ed8df3ea6d5e0a8c5a7defc56505532e215b module="containerd/tasks"
time="2018-05-03T13:18:12.455765863Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 3606.677118] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3607.809325] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3607.975205] aufs au_opts_verify:1597:dockerd[10370]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:18:17Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/67acef3ec8ff73639a8d36133a2c19be84d20236c9d71776a98847a541be732e/shim.sock" debug=false module="containerd/tasks" pid=29117
time="2018-05-03T13:18:20Z" level=info msg="shim reaped" id=67acef3ec8ff73639a8d36133a2c19be84d20236c9d71776a98847a541be732e module="containerd/tasks"
time="2018-05-03T13:18:20.470768902Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 3612.685662] aufs au_opts_verify:1597:dockerd[4502]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3612.913251] aufs au_opts_verify:1597:dockerd[4502]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3613.214634] aufs au_opts_verify:1597:dockerd[3406]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:18:22Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/36783f320c5e2ee7f90d0c92d9abf2c0812be019b1270e40e804b7c1753aa090/shim.sock" debug=false module="containerd/tasks" pid=29302
time="2018-05-03T13:18:23Z" level=info msg="shim reaped" id=36783f320c5e2ee7f90d0c92d9abf2c0812be019b1270e40e804b7c1753aa090 module="containerd/tasks"
time="2018-05-03T13:18:23.889381131Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 3615.976256] aufs au_opts_verify:1597:dockerd[10370]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3616.278263] aufs au_opts_verify:1597:dockerd[10370]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3616.332718] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:18:25Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/a4e96f4099a736507693cd40444c1dbf8129a2d058295a04a89c7bef3364f404/shim.sock" debug=false module="containerd/tasks" pid=29491
time="2018-05-03T13:18:27Z" level=info msg="shim reaped" id=a4e96f4099a736507693cd40444c1dbf8129a2d058295a04a89c7bef3364f404 module="containerd/tasks"
time="2018-05-03T13:18:27.220928709Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 3619.539129] aufs au_opts_verify:1597:dockerd[10464]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3619.782189] aufs au_opts_verify:1597:dockerd[10464]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3619.832195] aufs au_opts_verify:1597:dockerd[4216]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:18:29Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/48a80757c4692226a6ce67e1beb0542330c1b742e2fb7cb230df21db863a1afb/shim.sock" debug=false module="containerd/tasks" pid=29683
time="2018-05-03T13:18:30Z" level=info msg="shim reaped" id=48a80757c4692226a6ce67e1beb0542330c1b742e2fb7cb230df21db863a1afb module="containerd/tasks"
time="2018-05-03T13:18:30.669794951Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 3679.032472] aufs au_opts_verify:1597:dockerd[10464]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:19:28.174034845Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T13:34:50Z" level=info msg="shim reaped" id=631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e module="containerd/tasks"
time="2018-05-03T13:34:50Z" level=info msg="shim reaped" id=4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910 module="containerd/tasks"
time="2018-05-03T13:34:50.719177543Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:34:50.726367010Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 4603.089720] aufs au_opts_verify:1597:dockerd[4217]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4603.128367] aufs au_opts_verify:1597:dockerd[28416]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4603.157483] aufs au_opts_verify:1597:dockerd[28416]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4603.177768] aufs au_opts_verify:1597:dockerd[4217]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4603.202339] aufs au_opts_verify:1597:dockerd[4217]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4603.214106] aufs au_opts_verify:1597:dockerd[28416]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:34:52Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/da55011ecbcd7581d812b8a92d5072abb613b62b893a49e9b12238ad5363fda7/shim.sock" debug=false module="containerd/tasks" pid=30201
time="2018-05-03T13:34:52Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/358316942a7d744a69dcfdd34ca4beac0a737eebd3264cd349fd7b3c8ec3b5d0/shim.sock" debug=false module="containerd/tasks" pid=30202
time="2018-05-03T13:34:54Z" level=info msg="shim reaped" id=358316942a7d744a69dcfdd34ca4beac0a737eebd3264cd349fd7b3c8ec3b5d0 module="containerd/tasks"
time="2018-05-03T13:34:54Z" level=info msg="shim reaped" id=da55011ecbcd7581d812b8a92d5072abb613b62b893a49e9b12238ad5363fda7 module="containerd/tasks"
time="2018-05-03T13:34:54.669952996Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:34:54.671912740Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 4606.038885] aufs au_opts_verify:1597:dockerd[9510]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4606.074956] aufs au_opts_verify:1597:dockerd[4468]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4606.105321] aufs au_opts_verify:1597:dockerd[9510]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4606.133173] aufs au_opts_verify:1597:dockerd[9510]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4606.149764] aufs au_opts_verify:1597:dockerd[9510]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4606.159521] aufs au_opts_verify:1597:dockerd[4468]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:34:55Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/04f6c2f50cb959dfbd87e6482f1b57edcfd2de50ae59e989b59759d5c8ded896/shim.sock" debug=false module="containerd/tasks" pid=30556
time="2018-05-03T13:34:55Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/4b0c4ca425ae63aec65981b91049ded677e77a774a79ec42ab9cf82f58a53472/shim.sock" debug=false module="containerd/tasks" pid=30561
time="2018-05-03T13:34:55Z" level=info msg="shim reaped" id=4b0c4ca425ae63aec65981b91049ded677e77a774a79ec42ab9cf82f58a53472 module="containerd/tasks"
time="2018-05-03T13:34:55.741303482Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2018-05-03T13:34:55Z" level=info msg="shim reaped" id=04f6c2f50cb959dfbd87e6482f1b57edcfd2de50ae59e989b59759d5c8ded896 module="containerd/tasks"
time="2018-05-03T13:34:55.770089518Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
[ 4639.417246] aufs au_opts_verify:1597:dockerd[4217]: dirperm1 breaks the protection by the permission bits on the lower branch
time="2018-05-03T13:35:28.725226613Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T13:36:34.737944275Z" level=info msg="Processing signal 'terminated'"
We can see here that there were 5 containers that were created successfully. 4 of them each failed to start once (500 error code on the client) and then succeeded and one of them failed to start the first time with 500 error code and then failed to start 5 times with 403.
Here is the filtered version of the above logs with only the relevant parts:
time="2018-05-03T12:56:41.930879110Z" level=error msg="53fd32c54195dcb961a0c5eba182c0bd9088dd6c7043511e6f5b66153fec8469 cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T12:56:41.934737735Z" level=error msg="Handler for POST /containers/53fd32c54195dcb961a0c5eba182c0bd9088dd6c7043511e6f5b66153fec8469/start returned error: failed to create endpoint practical_fermi on network bridge: failed to save bridge endpoint d4455f1 to store: failed to update bridge store for object type *bridge.bridgeEndpoint: timeout"
time="2018-05-03T12:58:29.298547311Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T12:58:29.334262030Z" level=error msg="Handler for POST /containers/f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144/start returned error: failed to update the store state of sandbox: failed to update store for object type *libnetwork.sbState: timeout"
time="2018-05-03T13:05:24.922085385Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T13:05:24.981522266Z" level=error msg="4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910 cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T13:05:24.986465355Z" level=error msg="Handler for POST /containers/4929d8e75a56ca2e2d8abfe94f8245bd4f84a19309018b177df9db0c0125e910/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:385: running prestart hook 0 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\\\\\\\\\"2018-05-03T12:45:20Z\\\\\\\\\\\\\\\" level=fatal msg=\\\\\\\\\\\\\\\"failed to update store for object type *libnetwork.sbState: timeout\\\\\\\\\\\\\\\"\\\\\\\\n\\\\\\\"\\\"\": unknown"
time="2018-05-03T13:05:25.281302482Z" level=error msg="631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T13:05:25.281357469Z" level=error msg="Handler for POST /containers/631ca9eadfccdfc43f1537451f4c978a7c20bd7b792f8c1683c3018be1cd123e/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:385: running prestart hook 0 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\\\\\\\\\"2018-05-03T12:57:32Z\\\\\\\\\\\\\\\" level=fatal msg=\\\\\\\\\\\\\\\"failed to update store for object type *libnetwork.sbState: timeout\\\\\\\\\\\\\\\"\\\\\\\\n\\\\\\\"\\\"\": unknown"
time="2018-05-03T13:07:27.683705899Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T13:11:27.894633572Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T13:19:28.174034845Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
time="2018-05-03T13:35:28.725226613Z" level=error msg="f6f40b759a3e0132b56f8a6e3f13e509a148bda690446ba98825603600e71144 cleanup: failed to delete container from containerd: no such container"
Output of docker version:
Client:
Version: 17.12.1-ce
API version: 1.35
Go version: go1.9.4
Git commit: 7390fc6
Built: Tue Feb 27 22:17:40 2018
OS/Arch: linux/amd64
Server:
Engine:
Version: 17.12.1-ce
API version: 1.35 (minimum version 1.12)
Go version: go1.9.4
Git commit: 7390fc6
Built: Tue Feb 27 22:16:13 2018
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
It's not the exact machine on which this error occurred, but one with same configuration, i.e. difference is in number of containers and images, but those are also quite similar.
Containers: 34
Running: 2
Paused: 0
Stopped: 32
Images: 6
Server Version: 17.12.1-ce
Storage Driver: aufs
Root Dir: /mnt/docker/aufs
Backing Filesystem: extfs
Dirs: 119
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-1052-aws
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 36
Total Memory: 58.97GiB
Name: i-0d076c493fc5e0244
ID: RDNQ:6SAC:BZ4U:REOE:KTVI:DB3Z:RDJA:ZRA2:IUE6:GX5K:BFBR:EDOW
Docker Root Dir: /mnt/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.) AWS box
Related issues #235 is similar, but not sure if related
ping @ctelfer @fcrisciani - was anything around this fixed in a more recent version of Docker?
AFAICT, the only file that references sbState is sandbox_store.go. According to git blame this code has not been touched since July 2017 (on master, but don't think we'd change anything for a release branch that wasn't first in master).
@thaJeztah HI i see this error #8 0.312 runc run failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: failed to update store for object type *libnetwork.sbState: database not open
i am in the Docker version 27.1.2, build d01f264
Docker version 27.1.2, build d01f264 / # docker info Client: Version: 27.1.2 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.16.2 Path: /usr/local/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.2 Path: /usr/local/libexec/docker/cli-plugins/docker-compose
Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 27.1.2 Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353 runc version: v1.1.13-0-g58aa920 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 6.1.102-108.177.amzn2023.x86_64 Operating System: Alpine Linux v3.20 (containerized) OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 20.04GiB Name: runner-dind-api-wvkbk-runner-jkvbk ID: 5ce9b2d4-d083-4e1a-b114-1e71aa53e066 Docker Root Dir: /var/lib/docker Debug Mode: true File Descriptors: 24 Goroutines: 42 System Time: 2024-08-18T07:37:47.680443305Z EventsListeners: 0 Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine
Are you still seeing this issue on a current version of docker?
Sorry @thaJeztah , I missed this comment completely. It's been a while since I haven't run the types of workflows on EC2 instances that would cause this error to happen. Since there have been many releases in the meantime and nobody else complaining about this issue, I think we should close it 🙂 .
@noamgreen If you'd like, please go ahead and open a separate issue, as I don't think the root causes of these two are related.