connect icon indicating copy to clipboard operation
connect copied to clipboard

`kafka_franz` consumer offset does not reset based on `start_from_oldest`

Open bhataprameya opened this issue 2 years ago • 0 comments

  • Issue: In the kafka_franz input client, the consumer offset doesn't reset based on start_from_oldest after Kafka deletes data due to retention policy.

  • Current Solution: Restarting Benthos is required to reset the offset and resume consumption.

  • Expected Behavior: Consumer offset should automatically reset its offset based on start_from_oldest when the data it is trying to consume is no longer found in Kafka.

input config:

      kafka_franz:
        seed_brokers: ['${KAFKA_URL}']
        topics: ["${KAFKA_TOPIC}"]
        regexp_topics: false
        consumer_group: "${KAFKA_GROUP_ID}"
        client_id: benthos
        rack_id: --RACK_PLACE_HOLDER--
        checkpoint_limit: 50000
        commit_period: 5s
        start_from_oldest: false
        multi_header: false

bhataprameya avatar Feb 06 '24 20:02 bhataprameya