redpanda/kafka: add consumer groups committed offset timestamp metrics
our application must consume all messages, we can't afford to loose one by cleanup retention. Alongside with disk usage metrics, we need to measure the age of the lag to measure, time wise, the lateness we are having on each topic/partition.
Backports Required
- [x] none - not a bug fix
- [ ] none - this is a backport
- [ ] none - issue does not exist in previous branches
- [ ] none - papercut/not impactful enough to backport
- [ ] v23.3.x
- [ ] v23.2.x
- [ ] v23.1.x
Release Notes
Improvements
- add
redpanda_kafka_consumer_group_committed_offset_timestamp_secondsandredpanda_kafka_consumer_group_committed_offset_age_secondspublic metrics
I also wanted to add a metric that returns the current lag of a consumer group: redpanda_kafka_consumer_group_lag but I didn't find an easy way to do it. I can only easily get the topic name and the partition number from the model::topic_partition variable.
but I would need the log-end-offset to calculate the LAG but I don't know how to get it from groupe_probe.h
@fatpat i marked this as a draft since there is a merge conflict. let us know if this is something that still should be addressed and if you're able to work with ben on addressing his feedback. thanks!