Jesse Yates
Jesse Yates
Running some back of the envelop calculations, I'm finding the following: Keeping an (offset(int64), timestamp (int64), partition (int32)) is 20 bytes. Let's assume a ["big" cluster would go up to...
Keeping state seems like the only way to make this performant. You would have to read the message at the offset to get the timestamp (no key-only fetch request in...
FWIW we actually looked pretty far into this and it doesn't fit very well with how Burrow works, given how much pulling from different places in the topics you have...
At my day job, i ended writing something to do this as a secondary application, since it would muck quite a bit with the burrow internals IIRC https://github.com/teslamotors/kafka-helmsman/tree/master/kafka_consumer_freshness_tracker I believe...
Agree that this matches the implementation in s3 downloader. Would it be possible to add some tests around this? At the very least some documentation on the interface would be...
And some docs on the downloader interface please?