reverse-proxy icon indicating copy to clipboard operation
reverse-proxy copied to clipboard
verified publisher icon microsoft

Memory leak with KubernetesClient

Open dpetrovych opened this issue 1 year ago • 2 comments

While running YARP as ingress controller we see overall raising pattern of memory consumption and frequent restarts due to hitting pod limits.

To Reproduce

Run release/2.1/samples/KubernetesIngress.Sample/Ingress sample ingress for a breif while (10-15 minutes). Get a GC Dump.

image

Issue has a signature / related to: https://github.com/kubernetes-client/csharp/issues/1123 Present in KubernetesClient.Basic v.9.0.28. Fixed in KubernetesClient.Basic v.10.0.1

Further technical details

  • YARP Version: 2.1.0
  • Platform/base image: mcr.microsoft.com/dotnet/aspnet:8.0.3-bookworm-slim-amd64

dpetrovych avatar May 31 '24 10:05 dpetrovych

I started using YARP v.2.2.0-preview.1 (where KubernetesClient is update to v.10) and pattern of GC dumps have changed. Problem is not fully resolved, service continues to increse RAM usage daily. Here is comparison though 26 hours: one of the most numerable objects is CancelationTokenSource.

image

Also not sure, if such report would indicate a cyclic dependecy that prevents garbage collection: image

dpetrovych avatar Jun 04 '24 14:06 dpetrovych

Thanks for raising the issue. It'd make sense for us to move to the more recent KubernetesClient version with the fix.

MihaZupan avatar Jun 11 '24 14:06 MihaZupan

Presumably this isn't an issue anymore? We've moved to 10.1.19 in https://github.com/dotnet/yarp/pull/2338 (December 2023), and we're currently referencing 15.0.1 https://github.com/dotnet/yarp/blob/a91669145d395eb33f16d4c397a90a51c735d1d0/eng/Versions.props#L34

MihaZupan avatar Feb 26 '25 00:02 MihaZupan