Provide a non-destructive way to clean up debris on the filesystem

[g-suraj]

Feature request description

Related to https://github.com/containers/podman/issues/3799, it looks as though podman leaves layers and other stuff on the filesystem. This cannot be inspected (via podman system df) or cleaned (via podman system prune). You can however see this with

$ podman unshare du -sh /tmp/containers/suraj
128G /tmp/containers/suraj

The proposed solution in the aforementioned issue for tidying up inaccessible/leaked layers in /containers/storage/ is podman system reset. This seems like a pretty nuclear option that will affect any podman builds in progress. This results in platforms utilising podman build having to work a lot harder.

Suggest potential solution

It'd be ideal if podman system prune could delete all these "leaked" layers that are not currently in use.

[rhatdan]

Do you have any idea how these remnants are being left behind? If you have a simple repeater to show Podman leaking, we would love to fix it.

[g-suraj]

Hi @rhatdan! I believe the remnants are caused by container image builds that are being SIGKILL'd. I've been able to reproduce it with the following setup:

Dockerfile:

FROM ubi8/ubi
RUN truncate -s 10G out
RUN sleep infinity

Script run.sh

#!/usr/bin/env bash
podman build -f Dockerfile -t podmanleaker &
sleep 10 && kill -9 $!

And finally...

$ podman --version
podman version 4.2.0

$ podman system df
TYPE           TOTAL       ACTIVE      SIZE        RECLAIMABLE
Images         10          1           5.349GB     5.247GB (98%)
Containers     0           0           0B          0B (0%)
Local Volumes  0           0           0B          0B (0%)

$ podman unshare du -sh /tmp/containers/rootless_user/
1.1G    /tmp/containers/rootless_user/

$ ./run.sh &>/dev/null

$ podman system df
TYPE           TOTAL       ACTIVE      SIZE        RECLAIMABLE
Images         10          1           5.349GB     5.247GB (98%)
Containers     0           0           0B          0B (0%)
Local Volumes  0           0           0B          0B (0%)

$ podman unshare du -sh /tmp/containers/rootless_user/overlay
1.2G    /tmp/containers/rootless_user/

Does that help? Repeatedly running run.sh will increase the size of this folder and podman system prune doesn't help here.

[rhatdan]

Does buildah containers show anything?

Could you try podman system prune --external

  --external             Remove container data in storage not controlled by podman

[g-suraj]

So after trying that prune command

$ podman system df
TYPE           TOTAL       ACTIVE      SIZE        RECLAIMABLE
Images         5           3           13.57GB     3.845GB (28%)
Containers     0           0           0B          0B (0%)
Local Volumes  0           0           0B          0B (0%)
$ podman unshare du -sh /tmp/containers/rootless_user
82G     /tmp/containers/rootless_user
$ podman system prune --external
$ podman unshare du -sh /tmp/containers/rootless_user
82G     /tmp/containers/rootless_user

So buildah containers does display a bunch of things (even after the prune):

$ buildah --version
buildah version 1.29.1 (image-spec 1.0.2-dev, runtime-spec 1.0.2-dev)
$ buildah containers | wc -l
23

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.