stackabletech
Default replication factor for internal offsets topic prevents users from reading from clusters that have less than three nodes.
When deploying Kafka clusters with less than three nodes the default value of 3 for offsets.topic.replication.factor prohibits users from reading any values from topics.
Writing works fine, if topics are created with few enough partitions - or auto-created which observes available broker count for the replication factor.
But when reading for the first time, Kafka internally tries to create the __consumer_offsets topic, and this is required to have three partitions by default. And until this has been created no read requests are allowed.
The broker simply keeps logging
kafka [2023-05-05 15:20:39,078] INFO [Admin Manager on Broker 1001]: Error processing create topic request CreatableTopic(name='__consumer_offsets', numPartitions=50, replicationFactor=3, assignments=[], configs=[CreateableTopicConfig(name='compression.type', value='producer'), CreateableTopicConfig(name='cleanup.policy', value='compact'), CreateableTopicConfig(name='segment.bytes'
kafka org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.
Possible solutions are:
- documentation - this is not strictly speaking a bug, but also not nice
- catch it during validation in the operator and log an error
- automatically set this in the config when deploying less than 3 brokers
- ...
Good catch.
Technical question: Let's say a customer starts with 1 node, we set the setting to 1 as well. Customer later scales up to 5 nodes, we set it to 3. What will happen with existing topics? They will keep the original 1 setting, correct?
This should change the replication factor to three at that point. I'd have to test it to be 100% sure, but am pretty sure that this setting is actively monitored by Kafka, since it only applies to one topic specifically.
Just to make sure I don't create confusion, this setting only applies to the __consumer_offsets topic, this is not the default replication factor setting that applies to all new topics that are created without specifying a replication factor.
This at least needs some documentation but I'd prefer an automatic setting change I believe