cli `docker system prune -af --volumes` does not completely clear out overlay2 folder

Description

We run a lot of docker containers in our CI/CD, the majority using the docker run --rm option if that matters. We also build a few images and perform a docker system prune -af --volumes each night. Every couple of months we end up with 0 bytes remaining disk space due to /var/lib/docker/overlay2 retaining data despite the nightly cleanup. The only way to recover the disk space without breaking docker that I could find so far is a complete reinstall.

For reference:

$ docker system prune --all --force --volumes
Total reclaimed space: 0B
$ docker ps --all
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
$ docker volume ls
DRIVER    VOLUME NAME

# du -ch /var/lib/docker --max-depth=1
4.0K    /var/lib/docker/runtimes
28K     /var/lib/docker/volumes
67G     /var/lib/docker/overlay2
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/swarm
11M     /var/lib/docker/buildkit
4.0K    /var/lib/docker/containers
106M    /var/lib/docker/image
16K     /var/lib/docker/plugins
68K     /var/lib/docker/network
67G     /var/lib/docker
67G     total

# df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   62G   46G   13G  79% /

# ls -lh /var/lib/docker/overlay2/
total 5.6M
drwx--x--- 5 root root 4.0K Oct 19 16:24 0042cff5c2b1153d4e2cc8d1249433c3fdf40a9af374de7372e0bb487fa59a0e
drwx--x--- 4 root root 4.0K Oct 19 16:24 0042cff5c2b1153d4e2cc8d1249433c3fdf40a9af374de7372e0bb487fa59a0e-init
drwx--x--- 4 root root 4.0K Oct 31 09:03 00656fdea360c00da5d9549adcb5f968ab19280e6dde08892f6d26901ba84cca
drwx--x--- 4 root root 4.0K Oct 30 12:47 0081118c886142094d4bfdac3051331b04a5937bc0cd9db5f4d292a793d54a33
drwx--x--- 4 root root 4.0K Oct 19 14:41 0124757a1161c7b9bc1f4a725a5ef90c7d0b9598d31e53c8c59780fc686c2ca4
...
drwx--x--- 4 root root 4.0K Nov  7 08:47 x32c6n0iual0i5x438yhm91zc
drwx--x--- 4 root root 4.0K Nov  6 08:49 x9g9q67ib6r4pr799rkjcacir
drwx--x--- 4 root root 4.0K Nov  9 07:45 yo8bov1h8hqzcpt3ubdsj3de1

# ls -lh /var/lib/docker/overlay2/ | wc -l
1388

# apt-get purge docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras
# rm -rf /var/lib/docker
# rm -rf /var/lib/containerd
# apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   62G   20G   40G  33% /

Reproduce

Intensive use of docker run --rm, docker rmi and docker build over weeks
Run docker system prune -af --volumes every day overnight
Observe disk filling up over time

Expected behavior

After running docker system prune -af --volumes, all data from dangling images, completed containers etc should be removed

docker version

**Note:** Below o/p is collected _after_ a reinstall. The version running for the past months has been 24.0.6.

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:01 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:01 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.15.0-76-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.58GiB
 Name: ****
 ID: 2c71d842-867e-40fa-af3e-a6210ada7edd
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

Nov 11 '23 10:11 emike922

Best Option to Solve this issue

if the we running as docker service -> docker service update --force
you can check /var/lib/docker/overlay2 in linux terminal command "du -h --max-depth=1 /var/lib/docker/overlay2" to check maximum disk usage & check every docker process "docker inspect docker CONTAINER ID" or you can use docker disk space usage of Docker containers and images using command "docker system df"
docker system prune --all --force, caution if you are cleaning completely by removing the service you will lose the network & volumes.

Mar 15 '24 14:03 EyuelMathewos

I think i have the same issue. I don't have any

volumes
containers
images
networks

right now. But if i list the contents of /var/lib/docker:

4.0K    /var/lib/docker/swarm
19M     /var/lib/docker/buildkit
16K     /var/lib/docker/plugins
24M     /var/lib/docker/image
9.6G    /var/lib/docker/overlay2
8.0K    /var/lib/docker/tmp
1.5M    /var/lib/docker/containers
76K     /var/lib/docker/network
4.0K    /var/lib/docker/runtimes
34G     /var/lib/docker/volumes
44G     /var/lib/docker

I have tried every command i could find on the internet, nothing seems to help, including your commands (except for nuking the whole folder, but i want to avoid that if possible)

Is there a safe way to manually delete the contents of the /var/lib/docker folder?

Apr 22 '24 14:04 timon-michel-scopevisio

Have you tried docker builder prune? I'm pretty sure this is not handled with docker system prune`

Apr 22 '24 14:04 cpuguy83

Thanks for the proposal @cpuguy83. Unfortunately, docker builder prune -af has no additional effect - frees 0B. Any other suggestions? :-)

Apr 22 '24 15:04 emike922

@emike922 Did you, by chance, switch graphdrivers or even switch to the containerd backed image store?

Apr 22 '24 17:04 cpuguy83

I have changed the docker service to be rootless. After starting up the service as root again and cleaning up there, i have managed to clean my 44G down to 6G. After a quick sudo docker system prune --all --force (sudo or simply as root), it shows to have cleaned everything down to 18M (which is not completely clean, but that's alright)

So all in all - i understood that the rootless uses the data than the root-version, which doesn't seem to be the case. After thinking about it, it makes sense though. Maybe we can add a note to the documentation though, that you may want to clean up the installation before disabling the service? (here: https://docs.docker.com/engine/security/rootless/)

Apr 23 '24 09:04 timon-michel-scopevisio

Did you, by chance, switch graphdrivers or even switch to the containerd backed image store?

No, I didn't change anything, at least not consciously. We essentially run what is described at https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository.

Apr 23 '24 15:04 emike922

@emike922 if Container are running actively it will not clean any of use by docker system i think that why. i also recommend to run the command as root user also "sudo docker system prune --all --force" this command clean only unused containers data( containers with state stop ).

Apr 23 '24 17:04 EyuelMathewos

@EyuelMathewos I have made sure that no containers are running before running docker system prune -af, and running it as root made no difference either.

As an aside, why should I have to run the prune command as root? Our dockerd is already running as root. Is prune not executed with dockerd's permissions?

Apr 24 '24 07:04 emike922

@emike922 for checking the issue which docker dir is consuming alot of your space run "du -h --max-depth=1 /var/lib/docker" share the result and also check if there is no docker service running with replication 0?

Apr 24 '24 07:04 EyuelMathewos

@EyuelMathewos it's overlay2 that doesn't get pruned properly, please refer to my original logs in the issue description for the command output you requested.

How do I check for docker services with replication 0? Does the below o/p already answer the question implicitly?

$ docker service ls
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.

Apr 24 '24 08:04 emike922

@emike922 First Option check restarting docker service by "systemctl restart docker" Second Option "cd /var/lib/docker/overlay2" and "rm -r ./*" which remove the all overlay2 after that reboot the linux machine "reboot" & systemctl restart docker. this happend because all space is consumed and docker is not working properly i hope this helps.

Apr 24 '24 08:04 EyuelMathewos

As an aside, why should I have to run the prune command as root? Our dockerd is already running as root. Is prune not executed with dockerd's permissions?

I don't understand what you mean. In order to be able to communicate with the docker daemon, you should need to run it as the same user as the daemon. Or am i wrong?

Apr 24 '24 09:04 timon-michel-scopevisio

First Option check restarting docker service by "systemctl restart docker"

Interesting, that did clean up some of the data leftover in overlay2, but not all of it:

# docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
# docker system prune -af
Total reclaimed space: 12.77GB
# du -ch --max-depth=1 /var/lib/docker/
4.0K    /var/lib/docker/runtimes
28K     /var/lib/docker/volumes
12G     /var/lib/docker/overlay2
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/swarm
8.9M    /var/lib/docker/buildkit
4.0K    /var/lib/docker/containers
18M     /var/lib/docker/image
16K     /var/lib/docker/plugins
68K     /var/lib/docker/network
12G     /var/lib/docker/
12G     total
# systemctl restart docker
# du -ch --max-depth=1 /var/lib/docker/
4.0K    /var/lib/docker/runtimes
28K     /var/lib/docker/volumes
4.4G    /var/lib/docker/overlay2
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/swarm
8.9M    /var/lib/docker/buildkit
4.0K    /var/lib/docker/containers
18M     /var/lib/docker/image
16K     /var/lib/docker/plugins
68K     /var/lib/docker/network
4.4G    /var/lib/docker/
4.4G    total

Second Option "cd /var/lib/docker/overlay2" and "rm -r ./*" which remove the all overlay2 after that reboot the linux machine "reboot" & systemctl restart docker.

That may work, I can't try right now, but I can't really go around rebooting the host every couple of days. :( Ideally, docker system prune -af should catch and remove all of the stale/dangling entries in overlay2, wouldn't you agree?

I don't understand what you mean. In order to be able to communicate with the docker daemon, you should need to run it as the same user as the daemon. Or am i wrong?

I am running the docker CLI as non-privileged user in the "docker" group. As such, my user can access /var/run/docker.sock and therefore is able to issue commands like prune, which I expect will be executed by dockerd (which runs with root privileges). That's why I am wondering if my user really needs to be root (or use sudo), because dockerd is the one actually doing the pruning.

Apr 24 '24 12:04 emike922

@emike922 Okay that is progress let's try running & share the results it helps

docker volume ls
docker container ls -a
docker network ls
docker plugin ls
docker image ls

Apr 24 '24 12:04 EyuelMathewos

@EyuelMathewos I would say those all look unremarkable:

# docker volume ls
DRIVER    VOLUME NAME
# docker container ls -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
# docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
fe15e008745d   bridge    bridge    local
4890ede3afd1   host      host      local
2965bd8b6905   none      null      local
# docker plugin ls
ID        NAME      DESCRIPTION   ENABLED
# docker image ls
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

Apr 24 '24 13:04 emike922

@emike922 All are clean what remains in "/var/lib/docker/overlay2" try "ls -a /var/lib/docker/overlay2" share the results

Apr 24 '24 13:04 EyuelMathewos

@EyuelMathewos This is what's remaining. Can you infer anything from that?

# du -h --max-depth=1 /var/lib/docker/overlay2 | sort -h
28K     /var/lib/docker/overlay2/022076e4c2ea8a14ef639aada3bac355b796ea3364dc258cc82b5c7542ce16fa
28K     /var/lib/docker/overlay2/0fcd0396b0fb81670a13013a4bd423cee81dc283c1020d677f4593981eca5b4b
28K     /var/lib/docker/overlay2/108e5f2461abdce0c9f4859b7e612da9008589a1fe9bce8923cce5d4f522c5b4
28K     /var/lib/docker/overlay2/1904f568beb17ddc4111f6aafb97f0c05958c64553d9b1182d03a23758ba8dfa
28K     /var/lib/docker/overlay2/1972924ff48e6a6cd817f6a0646c4e2ee8f8f43f5e3767704ef81a47d707f535
28K     /var/lib/docker/overlay2/2be37bfa72549381a40e8a85668ddc518fc0a9d49ffce9896fa72595126fd06a
28K     /var/lib/docker/overlay2/2dca9ae8a227be68aea01e99833c6d2f83cc9fa077324fd9130c9a228c9e4752
28K     /var/lib/docker/overlay2/30d9ffe8f9d498881dc63f6aa19248dafce0ef6f2851ebfbb704f2a011df809c
28K     /var/lib/docker/overlay2/31e8024da2b4562eae3fd6a549e10e2115f2ed3bd79b16e23fed5bfbfab89c09
28K     /var/lib/docker/overlay2/338fbb0f1dd54f0a97e5be396c69b34e0cf8bffba9312b4357590d898283be6b
28K     /var/lib/docker/overlay2/3559251f7f5d8e9f1af7d20fd99a97eb3c3d10ed3064ce078d0f216dafb7ffe0
28K     /var/lib/docker/overlay2/37f5a25ae1cc76b33a9e3b0e3bebdf1deae64b4968047ad5e035ca6cb1efcf3a
28K     /var/lib/docker/overlay2/3fb9a6a43893ed3af74c4e8f452ffcd318c7e5826c88c08316059358da8216ff
28K     /var/lib/docker/overlay2/414bf51ab93ff4bb7e5670452e0aed11521f0d5e2d085933348b53790c8d5614
28K     /var/lib/docker/overlay2/43447517babe4472af38fa3d7819f20b4603731f7dd99e1c0a44476eb662a7be
28K     /var/lib/docker/overlay2/43be2a03435ec4a2afd535632bd77188f116e4066523e598230f0e692f6753de
28K     /var/lib/docker/overlay2/47045eb6f50f6e2c0c24bad3c306980be186fdcddd56a608c0d7304048be9c08
28K     /var/lib/docker/overlay2/4a0aebf5d367f5ca652e9322774ff0bb9ef9d90c512ef10b7871fe35f7de224c
28K     /var/lib/docker/overlay2/4aa6ba3ea0648a3ed73a12d28b9f9457244cecce98560f0dc4ef4bbdd812caa5
28K     /var/lib/docker/overlay2/4dc6b336af7b0b3cbca805bd484ea75f3ddc3b4e4282f60c570f3aba119551d9
28K     /var/lib/docker/overlay2/526fe45515f88fd9d3c3b4b7f8c9751a994cb0b612bdaa0465cf7edeb852d23d
28K     /var/lib/docker/overlay2/53569f132403fb412b4903da9df7ecc208b3ca6c9ab5ec5176c1dfba68be758f
28K     /var/lib/docker/overlay2/5922c49234bfa9f8eed1c6ad4ab18bdb5b7cf62e21984d2b792f7583fba12608
28K     /var/lib/docker/overlay2/5da93877906f89ee681d2a9bdd75486b84fed841a338cfcac592fb860e1ac165
28K     /var/lib/docker/overlay2/5e3697ef7973751d3e4a36c0c2ac8ee49f9db2cb0a5496279a3269595cbb06e1
28K     /var/lib/docker/overlay2/6267cc8c93157afe27b0664d44931a1e37b431c1bc12b4b2b73b26d172cbeaad
28K     /var/lib/docker/overlay2/72c4ea6eb9de00560135c525814260f9da34d5057ba7674e873725d0145e9d1f
28K     /var/lib/docker/overlay2/7556694aa0e4f86481a028f3ca87fc347d290ae013fd009780e4ef06007082b3
28K     /var/lib/docker/overlay2/75d336cf76495fb05be12d0a5cb198bf19e8c38cf872b8239796b8fe752e9b8f
28K     /var/lib/docker/overlay2/76789d56b87543c07948bd8b0e760402f4ce89027fb943c7753cd10db44d7076
28K     /var/lib/docker/overlay2/7ceba0d6be3b903574b012ae9e4122c8c26befc0a07bd6634197dfe39bb2e1ec
28K     /var/lib/docker/overlay2/7ec76091da6e3be065f2d0a41ce3cb6f321406292976b6b34f1ef8f00d2a7499
28K     /var/lib/docker/overlay2/7ef50662610cb28764440228696a6778e5bb17777f6864441fd2ecfd815aea97
28K     /var/lib/docker/overlay2/803b84975bf7b8a9e5e190360f227763a2e2aabbffe199f342773531200fd94c
28K     /var/lib/docker/overlay2/809a5cad8f0b658d035157b5736bf76195d9c06485a03a6f1fa962e82f0faa51
28K     /var/lib/docker/overlay2/8f221ee08f7e839a5210bad804f2de9df3e4da10642208afea1ac1c50d0e2953
28K     /var/lib/docker/overlay2/9147c8e51e9043768165cb9f65dfbbaeda4fee7f7f5606d7924ea34fe82823dc
28K     /var/lib/docker/overlay2/91490331cd32b72813ac3c5c7df2db6f7f1dee078592e5edb8a6fce1e3d0ad24
28K     /var/lib/docker/overlay2/918de3a940cd106c480d97aaff2504ecec62365b4838010f5e737cda5248eeb2
28K     /var/lib/docker/overlay2/9220c2bc4c265e1ca8a100a04ab56e21d821428839d840a95239dda2d9fb07f8
28K     /var/lib/docker/overlay2/9507a82ab080a827690bc1c6d764148bfe898bddc10372345eb8d5ad38454bb4
28K     /var/lib/docker/overlay2/97a57ca48d588a1e831a183a9323531410fc5d78ac94a1550d8a5e9ed8d248fb
28K     /var/lib/docker/overlay2/97b793f9325621eeb1a0c10cc18b9b63477edb7f46e8846af8124d1fe831bc7e
28K     /var/lib/docker/overlay2/9b380e9d350530581c7d01aa63e3bedeefed7682f6e7cfd5d92ad305c8826086
28K     /var/lib/docker/overlay2/a4eb0dfa3181c9429b5fc969f6e08e42ba133d0a340b40dad98f70341e7d710d
28K     /var/lib/docker/overlay2/a6471fdd4b709128a7db6ec3bf7ab74864f3ac346dbf31a2cbc74083a37100ba
28K     /var/lib/docker/overlay2/aac30f188a7b70dd8ceb9105f079bad9b6f7fb2e8d21473e72dfefce4144c093
28K     /var/lib/docker/overlay2/ab45b2a3a41e6df3ac90609cec4a5e3beacd5471c70e1c21a8307cd32c91b08c
28K     /var/lib/docker/overlay2/b2284c477db1d38995ecb55786bb7de10b7cab3c7e486accbfbb48be61cc8ac4
28K     /var/lib/docker/overlay2/b539972b90b41cd527d0cff6f78928a5029ee614c5a069c0019321d58c908de0
28K     /var/lib/docker/overlay2/b56fe94e35916b5ab6f1bb86b9c6466c2c18eadbcfafffd75e7b86eca4c2ab1f
28K     /var/lib/docker/overlay2/bc6d65968d77e5966785e85d3512b82bfc698d3dfe4e59b4ff70f78305289b07
28K     /var/lib/docker/overlay2/c136dbe94c420a1f2ce05add152dbb96d80a309513c43c87106016be9568573f
28K     /var/lib/docker/overlay2/c2a7d137110f29cc13953c1ef0094f25c4c161d3b449b63fb2beb6b5c89936b4
28K     /var/lib/docker/overlay2/c60f1691b727df2a249e37518b995df6a7ff0fd06e112391e3bd331dcc45028a
28K     /var/lib/docker/overlay2/ca82888ab4baa4e493dfe66f7f976892d597d538738b74ca076d76dc505f81cc
28K     /var/lib/docker/overlay2/cc1ea6b3cd19fcc8f6d1470135b45f2706056d18838ee7b2c78aeb269c2f4750
28K     /var/lib/docker/overlay2/ccdc55d46e440a23454c5b5e7878eebe46fd410e91fb83a54a516c32d6e9134c
28K     /var/lib/docker/overlay2/cf4783fec91839be29d9d4b4bda86d2d3535ca33067bc5a7e2891b5f32ec017a
28K     /var/lib/docker/overlay2/d1f3986e914b3bd28570665d4db2c9ca1747d1e0379d2f7a3ab25bf38a3fd861
28K     /var/lib/docker/overlay2/d2071e1c19daa0fc5683bd33bec80d278b238928fc11a8dd807d92bee92430c7
28K     /var/lib/docker/overlay2/e1143a7af864458600d0352f3b9be8f4d9a93b0435522abcfd00f6d08f4a4669
28K     /var/lib/docker/overlay2/e164d963bf77f593657c3c2a0821545ecdda2142d9ef08692b515695164354df
28K     /var/lib/docker/overlay2/e4f20f3382f7ddd6677fa6aae6979a0acdc6af27631f4980eecf23f7ca48782f
28K     /var/lib/docker/overlay2/ed7ed01076698aa79eadb2d08d09ebe4c0213e7e7298d4a8ffe1af05086160b3
28K     /var/lib/docker/overlay2/f2e53971f499eeb8bc26b9b6615ecd27a23bb2133b2c937eda811fb727dbbdf9
28K     /var/lib/docker/overlay2/f48de88276d01a858d7e77077950b9fc26f63bac545cf9e7b52dcffa4122f027
28K     /var/lib/docker/overlay2/f5ce3a39158839b60cf9a4a40d9c7c2d034df5492c5b633375a8c4428d44bb56
40K     /var/lib/docker/overlay2/022076e4c2ea8a14ef639aada3bac355b796ea3364dc258cc82b5c7542ce16fa-init
40K     /var/lib/docker/overlay2/0fcd0396b0fb81670a13013a4bd423cee81dc283c1020d677f4593981eca5b4b-init
40K     /var/lib/docker/overlay2/108e5f2461abdce0c9f4859b7e612da9008589a1fe9bce8923cce5d4f522c5b4-init
40K     /var/lib/docker/overlay2/1904f568beb17ddc4111f6aafb97f0c05958c64553d9b1182d03a23758ba8dfa-init
40K     /var/lib/docker/overlay2/1972924ff48e6a6cd817f6a0646c4e2ee8f8f43f5e3767704ef81a47d707f535-init
40K     /var/lib/docker/overlay2/2be37bfa72549381a40e8a85668ddc518fc0a9d49ffce9896fa72595126fd06a-init
40K     /var/lib/docker/overlay2/2dca9ae8a227be68aea01e99833c6d2f83cc9fa077324fd9130c9a228c9e4752-init
40K     /var/lib/docker/overlay2/30d9ffe8f9d498881dc63f6aa19248dafce0ef6f2851ebfbb704f2a011df809c-init
40K     /var/lib/docker/overlay2/31e8024da2b4562eae3fd6a549e10e2115f2ed3bd79b16e23fed5bfbfab89c09-init
40K     /var/lib/docker/overlay2/338fbb0f1dd54f0a97e5be396c69b34e0cf8bffba9312b4357590d898283be6b-init
40K     /var/lib/docker/overlay2/3559251f7f5d8e9f1af7d20fd99a97eb3c3d10ed3064ce078d0f216dafb7ffe0-init
40K     /var/lib/docker/overlay2/37f5a25ae1cc76b33a9e3b0e3bebdf1deae64b4968047ad5e035ca6cb1efcf3a-init
40K     /var/lib/docker/overlay2/3fb9a6a43893ed3af74c4e8f452ffcd318c7e5826c88c08316059358da8216ff-init
40K     /var/lib/docker/overlay2/414bf51ab93ff4bb7e5670452e0aed11521f0d5e2d085933348b53790c8d5614-init
40K     /var/lib/docker/overlay2/43447517babe4472af38fa3d7819f20b4603731f7dd99e1c0a44476eb662a7be-init
40K     /var/lib/docker/overlay2/43be2a03435ec4a2afd535632bd77188f116e4066523e598230f0e692f6753de-init
40K     /var/lib/docker/overlay2/47045eb6f50f6e2c0c24bad3c306980be186fdcddd56a608c0d7304048be9c08-init
40K     /var/lib/docker/overlay2/4a0aebf5d367f5ca652e9322774ff0bb9ef9d90c512ef10b7871fe35f7de224c-init
40K     /var/lib/docker/overlay2/4aa6ba3ea0648a3ed73a12d28b9f9457244cecce98560f0dc4ef4bbdd812caa5-init
40K     /var/lib/docker/overlay2/4dc6b336af7b0b3cbca805bd484ea75f3ddc3b4e4282f60c570f3aba119551d9-init
40K     /var/lib/docker/overlay2/526fe45515f88fd9d3c3b4b7f8c9751a994cb0b612bdaa0465cf7edeb852d23d-init
40K     /var/lib/docker/overlay2/53569f132403fb412b4903da9df7ecc208b3ca6c9ab5ec5176c1dfba68be758f-init
40K     /var/lib/docker/overlay2/5922c49234bfa9f8eed1c6ad4ab18bdb5b7cf62e21984d2b792f7583fba12608-init
40K     /var/lib/docker/overlay2/5da93877906f89ee681d2a9bdd75486b84fed841a338cfcac592fb860e1ac165-init
40K     /var/lib/docker/overlay2/5e3697ef7973751d3e4a36c0c2ac8ee49f9db2cb0a5496279a3269595cbb06e1-init
40K     /var/lib/docker/overlay2/6267cc8c93157afe27b0664d44931a1e37b431c1bc12b4b2b73b26d172cbeaad-init
40K     /var/lib/docker/overlay2/72c4ea6eb9de00560135c525814260f9da34d5057ba7674e873725d0145e9d1f-init
40K     /var/lib/docker/overlay2/7556694aa0e4f86481a028f3ca87fc347d290ae013fd009780e4ef06007082b3-init
40K     /var/lib/docker/overlay2/75d336cf76495fb05be12d0a5cb198bf19e8c38cf872b8239796b8fe752e9b8f-init
40K     /var/lib/docker/overlay2/76789d56b87543c07948bd8b0e760402f4ce89027fb943c7753cd10db44d7076-init
40K     /var/lib/docker/overlay2/7ceba0d6be3b903574b012ae9e4122c8c26befc0a07bd6634197dfe39bb2e1ec-init
40K     /var/lib/docker/overlay2/7ec76091da6e3be065f2d0a41ce3cb6f321406292976b6b34f1ef8f00d2a7499-init
40K     /var/lib/docker/overlay2/7ef50662610cb28764440228696a6778e5bb17777f6864441fd2ecfd815aea97-init
40K     /var/lib/docker/overlay2/803b84975bf7b8a9e5e190360f227763a2e2aabbffe199f342773531200fd94c-init
40K     /var/lib/docker/overlay2/809a5cad8f0b658d035157b5736bf76195d9c06485a03a6f1fa962e82f0faa51-init
40K     /var/lib/docker/overlay2/8f221ee08f7e839a5210bad804f2de9df3e4da10642208afea1ac1c50d0e2953-init
40K     /var/lib/docker/overlay2/9147c8e51e9043768165cb9f65dfbbaeda4fee7f7f5606d7924ea34fe82823dc-init
40K     /var/lib/docker/overlay2/91490331cd32b72813ac3c5c7df2db6f7f1dee078592e5edb8a6fce1e3d0ad24-init
40K     /var/lib/docker/overlay2/918de3a940cd106c480d97aaff2504ecec62365b4838010f5e737cda5248eeb2-init
40K     /var/lib/docker/overlay2/9220c2bc4c265e1ca8a100a04ab56e21d821428839d840a95239dda2d9fb07f8-init
40K     /var/lib/docker/overlay2/9507a82ab080a827690bc1c6d764148bfe898bddc10372345eb8d5ad38454bb4-init
40K     /var/lib/docker/overlay2/97a57ca48d588a1e831a183a9323531410fc5d78ac94a1550d8a5e9ed8d248fb-init
40K     /var/lib/docker/overlay2/97b793f9325621eeb1a0c10cc18b9b63477edb7f46e8846af8124d1fe831bc7e-init
40K     /var/lib/docker/overlay2/9b380e9d350530581c7d01aa63e3bedeefed7682f6e7cfd5d92ad305c8826086-init
40K     /var/lib/docker/overlay2/a4eb0dfa3181c9429b5fc969f6e08e42ba133d0a340b40dad98f70341e7d710d-init
40K     /var/lib/docker/overlay2/a6471fdd4b709128a7db6ec3bf7ab74864f3ac346dbf31a2cbc74083a37100ba-init
40K     /var/lib/docker/overlay2/aac30f188a7b70dd8ceb9105f079bad9b6f7fb2e8d21473e72dfefce4144c093-init
40K     /var/lib/docker/overlay2/ab45b2a3a41e6df3ac90609cec4a5e3beacd5471c70e1c21a8307cd32c91b08c-init
40K     /var/lib/docker/overlay2/b2284c477db1d38995ecb55786bb7de10b7cab3c7e486accbfbb48be61cc8ac4-init
40K     /var/lib/docker/overlay2/b539972b90b41cd527d0cff6f78928a5029ee614c5a069c0019321d58c908de0-init
40K     /var/lib/docker/overlay2/b56fe94e35916b5ab6f1bb86b9c6466c2c18eadbcfafffd75e7b86eca4c2ab1f-init
40K     /var/lib/docker/overlay2/bc6d65968d77e5966785e85d3512b82bfc698d3dfe4e59b4ff70f78305289b07-init
40K     /var/lib/docker/overlay2/c136dbe94c420a1f2ce05add152dbb96d80a309513c43c87106016be9568573f-init
40K     /var/lib/docker/overlay2/c2a7d137110f29cc13953c1ef0094f25c4c161d3b449b63fb2beb6b5c89936b4-init
40K     /var/lib/docker/overlay2/c60f1691b727df2a249e37518b995df6a7ff0fd06e112391e3bd331dcc45028a-init
40K     /var/lib/docker/overlay2/ca82888ab4baa4e493dfe66f7f976892d597d538738b74ca076d76dc505f81cc-init
40K     /var/lib/docker/overlay2/cc1ea6b3cd19fcc8f6d1470135b45f2706056d18838ee7b2c78aeb269c2f4750-init
40K     /var/lib/docker/overlay2/ccdc55d46e440a23454c5b5e7878eebe46fd410e91fb83a54a516c32d6e9134c-init
40K     /var/lib/docker/overlay2/cf4783fec91839be29d9d4b4bda86d2d3535ca33067bc5a7e2891b5f32ec017a-init
40K     /var/lib/docker/overlay2/d1f3986e914b3bd28570665d4db2c9ca1747d1e0379d2f7a3ab25bf38a3fd861-init
40K     /var/lib/docker/overlay2/d2071e1c19daa0fc5683bd33bec80d278b238928fc11a8dd807d92bee92430c7-init
40K     /var/lib/docker/overlay2/e1143a7af864458600d0352f3b9be8f4d9a93b0435522abcfd00f6d08f4a4669-init
40K     /var/lib/docker/overlay2/e164d963bf77f593657c3c2a0821545ecdda2142d9ef08692b515695164354df-init
40K     /var/lib/docker/overlay2/e4f20f3382f7ddd6677fa6aae6979a0acdc6af27631f4980eecf23f7ca48782f-init
40K     /var/lib/docker/overlay2/ed7ed01076698aa79eadb2d08d09ebe4c0213e7e7298d4a8ffe1af05086160b3-init
40K     /var/lib/docker/overlay2/f2e53971f499eeb8bc26b9b6615ecd27a23bb2133b2c937eda811fb727dbbdf9-init
40K     /var/lib/docker/overlay2/f48de88276d01a858d7e77077950b9fc26f63bac545cf9e7b52dcffa4122f027-init
40K     /var/lib/docker/overlay2/f5ce3a39158839b60cf9a4a40d9c7c2d034df5492c5b633375a8c4428d44bb56-init
848K    /var/lib/docker/overlay2/l
12M     /var/lib/docker/overlay2/0f6eec1f33ff745a875c6da25cc63192211ed7cde6f6e1fa88a144e5efb61258
12M     /var/lib/docker/overlay2/11bd070e718a4ad3a5fd8b95b2ad3dee8bd194f1fda9aa47a7b6a0ee13360996
12M     /var/lib/docker/overlay2/12e0c9266f57ab7d188939bd0caf22e20acc0ea95b114e7077b4c06e10546e2a
12M     /var/lib/docker/overlay2/14a3ec226cf924ae8d53c0dab26000782ca661d1f0c3213713d6976355e4a2a0
12M     /var/lib/docker/overlay2/18038f74bf029813c2d6a5d025efcff69c027d9dd2e0a686c052a6bcb4a14820
12M     /var/lib/docker/overlay2/1d5f2d45079dd5947b54c348fd79c612bf25b760853e309acd25f640a6624246
12M     /var/lib/docker/overlay2/22aafcfb818ebcc579f26473df50d878f81874015c598852224300d27806e11a
12M     /var/lib/docker/overlay2/32b886491d2a86ceec1997149be442ad1e83d0a5d9e475dcf1ecc95c314f3df9
12M     /var/lib/docker/overlay2/339ce9be6c0ecb84d6c4c470ae22107a1cfd993d205cd569f80d4317664698ce
12M     /var/lib/docker/overlay2/3507739e44bb87b5a0d5f3ef6a5fbd8faa760bd189949b529f1bc7e21ed94cdd
12M     /var/lib/docker/overlay2/3a3145821215e2aeabff60761f011a38ed79de590b23eb651797299b4c22f5f9
12M     /var/lib/docker/overlay2/447c1f1c9c2a162b7a12f4804ea7772acabd187fad6a93e741c4af43bb1527a0
12M     /var/lib/docker/overlay2/487fe220f27259b9ffc07d6bb86a1234c6d10682c7fcb18d14bc003717a39ab1
12M     /var/lib/docker/overlay2/4a2775b006d77a1a0b5861fff51007ed1ad4e043261aae82bac400e4fc3b71c4
12M     /var/lib/docker/overlay2/4d7820155509f96d95d1c2a4e328cb0c6d714de1575ea6b69dbacf387bcaa14b
12M     /var/lib/docker/overlay2/500707b675fc231c8a313ffb1afd1bbd9da73f20bce9395b18dbdcc675d204d2
12M     /var/lib/docker/overlay2/5e1b2c4bbbfd51c93fcf23baeb66be6023173e3eda06436581cbb0d156eef224
12M     /var/lib/docker/overlay2/703d8ba6155597f72fd1c5ce356634dca7099f38fa5ebf1b226b10f4b3b35a15
12M     /var/lib/docker/overlay2/71a0d1a35aa8dfabf9981f2354fbcfe5aed51b733a9918031b1ce7b2cc985571
12M     /var/lib/docker/overlay2/8364866b9f7539541ac2ce3532f98aa01c74610ab1d42d6ddc1dd782f01df777
12M     /var/lib/docker/overlay2/8ab0ab17cc229de1c6268c824deb17d10d697262e2c06a6d249f7e3a3c3ffad6
12M     /var/lib/docker/overlay2/9414481a8cc0dc7be44ae674627d0ccd0a6448230e0abcd927e9dce2cad2b117
12M     /var/lib/docker/overlay2/a98c079a4e95b8d73eb5090846863bc1f37b432cf808ee47e3948e175ac2669e
12M     /var/lib/docker/overlay2/b5f2c0d455769d041cce8c7c8f2f1aeeb177ee226644a9b5576a8497e29a8da1
12M     /var/lib/docker/overlay2/bcd58aab586d43bf42b2de42eb33e0d110e80b359c1aa0380d468f74e94d814f
12M     /var/lib/docker/overlay2/cc4e13d5d1e0a70d3b777925566825a3eccc4d338e659c06e222a14282b2a071
12M     /var/lib/docker/overlay2/d10459fbe32625f99251269cc1ee18ccddf246d4f3a1db5ab4c8fc81f6083df8
12M     /var/lib/docker/overlay2/d53d571be663ad78f04de744e78c23107aa9c647a2abdc312f3de1b52214f437
12M     /var/lib/docker/overlay2/d982a71269464169f40dd9b97165aa65f5d905b5b38ba142ba1a0ae2c2814f33
12M     /var/lib/docker/overlay2/dece2e0ebe3c4be2b977148311878680dfc2494419aa42405f4d92f3f3131514
12M     /var/lib/docker/overlay2/eb6ade431731747c02694951e965fd87c259afe626d9ef2ab1135978f0242f94
12M     /var/lib/docker/overlay2/eb774ed51705a1e2b9d0a05fb4c9e9cbf7fa28e65936f888e11efaff4238db92
12M     /var/lib/docker/overlay2/f3bee4679e4ba49ef1448dd4dc7f911239d93ef0bb4dac99fc30042cf8c1dc0d
12M     /var/lib/docker/overlay2/ff684ec445f579fabcc5f948d643efc22f7de71748d5841d2a5d230217865632
43M     /var/lib/docker/overlay2/c6b098a51cf9ecd3fadbdfe7e4595969a897d158ef50217adf9dd045ee244792
118M    /var/lib/docker/overlay2/114f461ddc4a8dc93baebfaeae6b99df6ed2af6e1ce139d40aa2d6b3570bd972
118M    /var/lib/docker/overlay2/1bdaa1e424e599e988066acd1187b07bc199ed37f91c1179d4a01d200f37720d
118M    /var/lib/docker/overlay2/1e1a587688a9f9e3691f02851dda6196c04d0a3f8c999af5c191db4cfab76a9b
118M    /var/lib/docker/overlay2/24e5eb43bd4d7100b1956011acb5d35a6771f915a53069eebd8d62c8fa216ecc
118M    /var/lib/docker/overlay2/284c5bfaf3217d18abe21e596fef954a412e2b49c38b3fae3c1891fcc632636a
118M    /var/lib/docker/overlay2/2b31b050d250d6fc23f6c3bae706e871debba9c1ff86f6a1c2d0bdd39c8d42ab
118M    /var/lib/docker/overlay2/39364c897e59457c6ac627a4ed65fc0b6d7d5292fcc0ff70afe0c64baeda3254
118M    /var/lib/docker/overlay2/426bb0fddab6257b7fdf13fe764f4d01810779eb5f7464965c369b2222b823fc
118M    /var/lib/docker/overlay2/446e7eb52b903547fd3f94206a54c1fcc3ca6bcf55b68b53d89be46acd012692
118M    /var/lib/docker/overlay2/486162e92862c37bf6a9392a7af340b9c050a83990193abb06d6520fb5861406
118M    /var/lib/docker/overlay2/51bdc4f2a842b3c4b8fd9336deffa226de20696304251e3434aef8f98cb02d62
118M    /var/lib/docker/overlay2/5fc85c3e2f3e6abd178ce61768434f9d842fa94e946cf6d6f9030b0645e0aa2a
118M    /var/lib/docker/overlay2/64eab9f58bc9b4e5e88f3f3fc0351cbe3406f3b4cdc21bcd52c8ea4e22ddf26e
118M    /var/lib/docker/overlay2/67302d6cc7028e02964400c76e235380e78ac8a4ae01ece598f566b35f1e8f75
118M    /var/lib/docker/overlay2/7868200c332a8a61c020e152b2b7fc8ad9b4a6efb7fdf957d1d07405aebfd046
118M    /var/lib/docker/overlay2/9da93202f6c38ef718a63f92760590fa44ff92eb457eb6906a30ce344016993d
118M    /var/lib/docker/overlay2/9eace8be6aa95e9dea7f045e4599a0356431fa3db7ebbe06c48163db21b20789
118M    /var/lib/docker/overlay2/a1da59e1f0e8faeb11c3bf00b0242160a6542d9d3def648082cf6e6d7ef791e0
118M    /var/lib/docker/overlay2/a4250fee2a542a56f71f3c27b676893277d64bb9bfa2cb2dee9486b99b1aa86e
118M    /var/lib/docker/overlay2/af320e7e38464f3c30f9f0f1e2430a7811ad549fd893a3eb4521d7770398fd26
118M    /var/lib/docker/overlay2/af66e4862b415f6a1a7bfc82291c0d147214f4e79ec2f548db8f7dd8ef10f8b6
118M    /var/lib/docker/overlay2/b164dce7d62ef01efdccd8ba9d41aa673222882242414964efac532b3c826637
118M    /var/lib/docker/overlay2/be5adaaefd075d71492de48d216baab4aa7d55ce0212f8b6c96bbef59a5d9a6e
118M    /var/lib/docker/overlay2/c37b262d2c42cc54ade4ba8c49940b0b30dd685d0f8dec6409163fda52c1807e
118M    /var/lib/docker/overlay2/cacf2b3a775594495996b2b977c6fd15b65ed0d7ef4ae97cf5c8a6c8dad0db81
118M    /var/lib/docker/overlay2/d8d54950701480994d5a7674f8d9d13862bd744952c21d88372d415a915c61eb
118M    /var/lib/docker/overlay2/e1514fc1d13639311aa4d2243bd16de3488c4027e0db84333bdad906bf3d70c5
118M    /var/lib/docker/overlay2/e841a346a27e857f15730468911efd3dc42dcc1e8f48ad04298d54e3456ad1db
118M    /var/lib/docker/overlay2/e8fb3a906f66e781e97aa15136c9197bdc1c5fb4b86bb89178de33d77c0d08dd
118M    /var/lib/docker/overlay2/e9145e0bfcdc0b5ad0980819dfbcaf0a7e6f11b76db7b6ac4d7d6f57151f7cbf
118M    /var/lib/docker/overlay2/efc83e03194a0a189868cfe1c3d9b552d71aad5b5340aa85ce9e8e21f7226a0a
118M    /var/lib/docker/overlay2/f165555f036fa67e0ace61a0b30a47339bce1eb03e835fe5441fadfac06eb578
118M    /var/lib/docker/overlay2/f3d8e6bf9be751e98ecf62a3c99ef0708b66129f3f11741a19d75d3506c09a93
118M    /var/lib/docker/overlay2/fdd9df4cb04e56078881befc7ba4c2ceaec391adf8b725ef5743c7c32f41f7d1
4.4G    /var/lib/docker/overlay2

Apr 24 '24 13:04 emike922

@emike922 we should list out where is max disk usage "du -h --max-depth=1 /var/lib/docker/overlay2"

Apr 24 '24 13:04 EyuelMathewos

@EyuelMathewos I updated the output in the previous comment with the information in an attempt to limit the length of this thread somewhat.

Apr 24 '24 13:04 emike922

@emike922 Okay that is helpful, let try

"systemctl stop docker"
"systemctl stop docker.socket"
check the status "systemctl status docker.socket"
after that then -> "cd /var/lib/docker/overlay2" and "rm -r ./*"

Apr 24 '24 13:04 EyuelMathewos

@emike922 Finally start

"systemctl start docker"
"systemctl start docker.socket"

Apr 24 '24 13:04 EyuelMathewos

@EyuelMathewos That certainly removed everything from overlay2, however, docker still seems to "know" about and attempt to use the deleted entries:

+ docker build --build-arg BASE_IMAGE_VERSION="1.2.3" --tag "target-image:1.0.0" .
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.87kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [auth] path/base-image:pull token for private-registry.example.com
#3 DONE 0.0s

#4 [internal] load metadata for private-registry.example.com/path/base-image:1.2.3
#4 DONE 0.5s

#5 [ 1/12] FROM private-registry.example.com/path/base-image:1.2.3@sha256:7b6850578fdb5bc31dd59ac0b10b4b131d201603cf680212e0e5b918a22eafcc
#5 resolve private-registry.example.com/path/base-image:1.2.3@sha256:7b6850578fdb5bc31dd59ac0b10b4b131d201603cf680212e0e5b918a22eafcc done
#5 sha256:5c13c8f19ac777c393e8916cd38d5a6b3b66e873e9d8692968452ba08dbe9159 1.70kB / 1.70kB done
#5 sha256:bab6eb336678c7c650680c940838514d2e58b5aab8b2a7f61bc889cd084d66c3 4.19MB / 31.45MB 0.1s
#5 sha256:7b6850578fdb5bc31dd59ac0b10b4b131d201603cf680212e0e5b918a22eafcc 741B / 741B done
#5 sha256:bab6eb336678c7c650680c940838514d2e58b5aab8b2a7f61bc889cd084d66c3 31.45MB / 31.45MB 0.3s
#5 sha256:bab6eb336678c7c650680c940838514d2e58b5aab8b2a7f61bc889cd084d66c3 31.45MB / 31.45MB 0.4s done
#5 extracting sha256:bab6eb336678c7c650680c940838514d2e58b5aab8b2a7f61bc889cd084d66c3
------
 > [ 1/12] FROM private-registry.example.com/path/base-image:1.2.3@sha256:7b6850578fdb5bc31dd59ac0b10b4b131d201603cf680212e0e5b918a22eafcc:
------
Dockerfile:4
--------------------
   2 |     ARG BASE_IMAGE_VERSION
   3 |     
   4 | >>> FROM ${BASE_IMAGE_NAME}:${BASE_IMAGE_VERSION}
   5 |     
   6 |     # Redacted comment
--------------------
ERROR: failed to solve: failed to register layer: open /var/lib/docker/overlay2/c6b098a51cf9ecd3fadbdfe7e4595969a897d158ef50217adf9dd045ee244792/.tmp-committed3238700635: no such file or directory

Apr 24 '24 13:04 emike922

@emike922 did you stop all docker linux services before deleting the overlay2 dir files ? "systemctl start docker"

"systemctl start docker.socket"
Try cleaning "docker builder prune --all --force"

Apr 24 '24 14:04 EyuelMathewos

@EyuelMathewos Sorry, I forgot to include the logs for that.. But yes, I did stop all Linux services as you suggested:

# systemctl stop docker
Warning: Stopping docker.service, but it can still be activated by:
  docker.socket
# systemctl stop docker.socket
# systemctl status docker.socket
● docker.socket - Docker Socket for the API
     Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset: enabled)
     Active: inactive (dead) since Wed 2024-04-24 15:37:48 CEST; 8s ago
   Triggers: ● docker.service
     Listen: /run/docker.sock (Stream)

Apr 22 18:07:38 node-10-210-155-101 systemd[1]: Starting Docker Socket for the API.
Apr 22 18:07:38 node-10-210-155-101 systemd[1]: Listening on Docker Socket for the API.
Apr 24 15:37:48 node-10-210-155-101 systemd[1]: docker.socket: Succeeded.
Apr 24 15:37:48 node-10-210-155-101 systemd[1]: Closed Docker Socket for the API.
# cd /var/lib/docker/overlay2
# rm -r ./*
# ls -lh
total 0
# systemctl start docker
# systemctl start docker.socket
# systemctl status docker.socket
● docker.socket - Docker Socket for the API
     Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-04-24 15:39:05 CEST; 32s ago
   Triggers: ● docker.service
     Listen: /run/docker.sock (Stream)
      Tasks: 0 (limit: 18991)
     Memory: 4.0K
     CGroup: /system.slice/docker.socket

Apr 24 15:39:05 node-10-210-155-101 systemd[1]: Starting Docker Socket for the API.
Apr 24 15:39:05 node-10-210-155-101 systemd[1]: Listening on Docker Socket for the API.

I also did a docker builder prune --all --force just now, but the build still produces the exact error shared before (complaining about /var/lib/docker/overlay2/c6b098a51cf9ecd3fadbdfe7e4595969a897d158ef50217adf9dd045ee244792). Do you have any other ideas how to make docker "forget" that it used to exist?

By the way, thanks a lot for your continued support on this! :)

Apr 24 '24 14:04 emike922

sudo systemctl stop docker sudo rm -rf /var/lib/docker/buildkit sudo systemctl start docker

Apr 24 '24 15:04 EyuelMathewos

I followed your steps, but the error is unfortunately completely unchanged..

Apr 24 '24 15:04 emike922

@emike922 remove all docker engine files begin with new by cleaning

"systemctl stop docker"
"systemctl stop docker.socket"
sudo rm -rf /var/lib/docker
"systemctl start docker"
"systemctl start docker.socket"

Apr 24 '24 18:04 EyuelMathewos

@EyuelMathewos That seems to have done it! Docker recovered after the restart and is no longer generating the above error during builds 🥳

Does any of that help to narrow down why docker system prune -af is not able to detect and remove all of the unused data? Ideally that should be the only command I need to run every now and then. For sure nuking the entire /var/lib/docker dir every time is a non-starter..

Apr 25 '24 07:04 emike922