Sergey Rustamov

Prod Heroic is dying frequently from OutOfMemoryError exceptions (a.k.a. OOM-ing)

2

Occasionally instances serving query requests will be restarted with such logs: _Terminating due to java.lang.OutOfMemoryError: Java heap space_ There is no any additional stack in the logs.

Add exponential backoff strategy for Elasticsearch writes

3

This strategy will allow smooth out any spikes on metrics ingestion pipeline.

This strategy will allow smooth out any spikes on metrics ingestion pipeline. Depending on Bigtable client this could be one option for the solution [BulkOptions.enableBulkMutationThrottling](https://cloud.google.com/bigtable/docs/hbase-client/javadoc/com/google/cloud/bigtable/config/BulkOptions.Builder.html#enableBulkMutationThrottling--)

Large UDP payloads can be silently dropped

4

Due to nature of UDP protocol data could be silently dropped if reached specific limits. As the data/metrics are dropped silently usually we alerted by teams noticing missing metrics. One...

This is related to this [PR](https://github.com/spotify/heroic/pull/686) issue where metrics won't be ingested to Bigtable due to row key length is above the limit of 4096 bytes. Preventing these kind of...