Prod Heroic is dying frequently from OutOfMemoryError exceptions (a.k.a. OOM-ing)

Open malish8632 opened this issue 5 years ago • 2 comments

Occasionally instances serving query requests will be restarted with such logs:

Terminating due to java.lang.OutOfMemoryError: Java heap space

There is no any additional stack in the logs.

Jul 09 '20 15:07 malish8632

We're profiling Heroic using JDK Mission Control 7.0.1+b01 and IntelliJ's inbuilt profiler.

We're basically exploring and trying to repro the OOM locally.

Jul 09 '20 16:07 sming

The plan now is to repro the OOM in a small cluster in prod using this JVM@GKE tip: https://danlebrero.com/2018/11/20/how-to-do-java-jvm-heapdump-in-kubernetes/ .

Jul 14 '20 16:07 sming