connect
connect copied to clipboard
`kafka_franz` consumer offset does not reset based on `start_from_oldest`
-
Issue: In the
kafka_franzinput client, the consumer offset doesn't reset based onstart_from_oldestafter Kafka deletes data due to retention policy. -
Current Solution: Restarting Benthos is required to reset the offset and resume consumption.
-
Expected Behavior: Consumer offset should automatically reset its offset based on
start_from_oldestwhen the data it is trying to consume is no longer found in Kafka.
input config:
kafka_franz:
seed_brokers: ['${KAFKA_URL}']
topics: ["${KAFKA_TOPIC}"]
regexp_topics: false
consumer_group: "${KAFKA_GROUP_ID}"
client_id: benthos
rack_id: --RACK_PLACE_HOLDER--
checkpoint_limit: 50000
commit_period: 5s
start_from_oldest: false
multi_header: false