arroyo
arroyo copied to clipboard
Reset DLQ buffers during rebalances
Today, as DLQ is currently implemented, the DLQ buffer is never "cleared" until consumer process is shut down. As a result, consumers will keep, in-memory, multiple partitions' worth of buffer data, if the consumers were assigned those partitions at any point in their lifetime. And, we know that during Kafka rebalancing, partitions are being shuffled around to various consumers.
This issue is for tracking the work to reset the buffers for specific partitions during partition revocation. The goal of this change is to ensure better memory management within the DLQ implementation of Arroyo.