arroyo icon indicating copy to clipboard operation
arroyo copied to clipboard

Reset DLQ buffers during rebalances

Open ayirr7 opened this issue 1 year ago • 0 comments

Today, as DLQ is currently implemented, the DLQ buffer is never "cleared" until consumer process is shut down. As a result, consumers will keep, in-memory, multiple partitions' worth of buffer data, if the consumers were assigned those partitions at any point in their lifetime. And, we know that during Kafka rebalancing, partitions are being shuffled around to various consumers.

This issue is for tracking the work to reset the buffers for specific partitions during partition revocation. The goal of this change is to ensure better memory management within the DLQ implementation of Arroyo.

ayirr7 avatar Dec 12 '24 07:12 ayirr7