stackabletech
Support deploying Kafka with KRaft instead of ZooKeeper for consensus building
In later versions Kafka supports running without ZooKeeper and provides its own KRaft consensus algorithm.
We should look into enabling the operator to deploy Kafka in this mode.
I just wanted to mention that according to https://www.confluent.io/blog/zookeeper-to-kraft-with-confluent-kubernetes/:
With Kafka 4.0 and Confluent Platform 8.0, Kafka will complete its journey to KRaft by removing the deprecated ZK mode,
So we need to tackle this at some point of time
I assume we want a CRD change to remove the Zookeeper ConfigMap reference.
There will be a CRD versioning discussion during the on-site about how to version CRDs (eg: how to choose from v1alpha2 vs v1beta1 vs v2alpha1).
Release Notes
KRaft is now available for all supported Stackable Kafka versions as an experimental preview to replace the Apache ZooKeeper quorum. Experimental support for Kafka 4 was added.
Breaking changes
- The broker
server.propertieswas renamed tobroker.properties. This will affect existing config overrides.
Whats missing / TODOs
- The broker/controller.properties file is currently copied to /tmp in order to add dynamic properties This could be improved by using an emptyDir as readWriteConfig.
- Currently the start commands are formated via Rust code, config-utils could be used instead
- There is no print of the properties file at startup. This could be readded if no sensitive information is leaked.
- Controller PDBs are currently set to 1. We could increase this to 3.
- No antiaffinities are currently set for controllers and brokers not ending up on the same node.
- Kafka 3.7.2 only supports a static quorum. This means no Controller scaling is possible.
- The Kafka docker image uses a very old broker.yaml for the JMX export rules. This should be checked and adapted for Kafka 4 (and possibly 3).
Do we have docs for this we can link to?
Yes, https://docs.stackable.tech/home/nightly/kafka/usage-guide/kraft-controller/