Podman fuse-overlyfs consumes 100% CPU

[maestropandy]

I have a mailman process which runs time to time, and creates lot of mails and logs. Where postfix incoming directories contains 10k files in average. When this happens, fuse-overlayfs uses 100% of the CPU and everything works very slowly. which results mail queued and delay at receiver end.

I have tried below things, but that doesn't help

- stop container 
- restart  sys-module-fuse.device sys-fs-fuse-connections.mount
- start container

That helped for a while to speeded mail queue 10x faster, later started to lagging, even "ls <10kfiles> response itself slow.

Here is result of top command

top - 03:42:08 up 61 days, 14 min, 5 users, load average: 29.51, 56.80, 56.23 Tasks: 599 total, 2 running, 597 sleeping, 0 stopped, 0 zombie %Cpu(s): 6.3 us, 0.2 sy, 0.0 ni, 93.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 79663.8 total, 1196.6 free, 28797.9 used, 49669.3 buff/cache MiB Swap: 6619.0 total, 6192.0 free, 427.0 used. 46874.8 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 278622 clamscan 20 0 1847904 1.2g 9588 S 100.3 1.6 111:39.59 clamd 75184 pandy 20 0 13.3g 13.3g 1188 R 100.0 17.1 558:53.75 fuse-overlayfs

Expected behaviour Fuse-overlayfs memory consumption shouldn't be above 80%

podman info

host: arch: amd64 buildahVersion: 1.23.1 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon-2.0.32-1.module+el8.5.0+20494+0311868c.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.32, commit: 152262b46a0d328c2746a648892a6b9a67922199' cpus: 32 distribution: distribution: '"ol"' variant: server version: "8.5" eventLogger: file hostname: pandy idMappings: gidmap: - container_id: 0 host_id: 1002 size: 1 - container_id: 1 host_id: 296608 size: 65536 uidmap: - container_id: 0 host_id: 1002 size: 1 - container_id: 1 host_id: 296608 size: 65536 kernel: 5.4.17-2136.306.1.3.el8uek.x86_64 linkmode: dynamic logDriver: k8s-file memFree: 990793728 memTotal: 83533500416 ociRuntime: name: runc package: runc-1.0.3-1.module+el8.5.0+20494+0311868c.x86_64 path: /usr/bin/runc version: |- runc version 1.0.3 spec: 1.0.2-dev go: go1.16.12 libseccomp: 2.5.1 os: linux remoteSocket: path: /run/user/1002/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.1.8-1.module+el8.5.0+20416+d687fed7.x86_64 version: |- slirp4netns version 1.1.8 commit: d361001f495417b880f20329121e3aa431a8f90f libslirp: 4.4.0 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.1 swapFree: 6492778496 swapTotal: 6940520448 uptime: 1464h 11m 31.53s (Approximately 61.00 days) plugins: log:

• k8s-file
• none
• journald network:
• bridge
• macvlan volume:
• local registries: search:
• container-registry.pandy.com
• docker.io store: configFile: /home/pandy/.config/containers/storage.conf containerStore: number: 1 paused: 0 running: 1 stopped: 0 graphDriverName: overlay graphOptions: overlay.mount_program: Executable: /usr/bin/fuse-overlayfs Package: fuse-overlayfs-1.8-1.module+el8.5.0+20494+0311868c.x86_64 Version: |- fusermount3 version: 3.2.1 fuse-overlayfs: version 1.8 FUSE library version 3.2.1 using FUSE kernel interface version 7.26 graphRoot: /u01/podman/pandy/containers/storage graphStatus: Backing Filesystem: extfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "false" imageStore: number: 1 runRoot: /run/user/1002/containers volumePath: /u01/podman/pandy/containers/storage/volumes version: APIVersion: 3.4.2 Built: 1644019375 BuiltTime: Sat Feb 5 00:02:55 2022 GitCommit: "" GoVersion: go1.16.12 OsArch: linux/amd64 Version: 3.4.2