Make Zookeeper usage optional for KafkaContainer

[sergey-morenets]

Hi

Starting since Kafka 2.8 Zookeeper is now an optional component because Kafka node can act as an active controller (new "process.roles" property). However I see from the code (configure method in the KafkaContainer class), that zookeeper.connect property is always added via environment variables:

if (externalZookeeperConnect != null) {
            withEnv("KAFKA_ZOOKEEPER_CONNECT", externalZookeeperConnect);
        } else {
            addExposedPort(ZOOKEEPER_PORT);
            withEnv("KAFKA_ZOOKEEPER_CONNECT", "localhost:" + ZOOKEEPER_PORT);
            command += "echo 'clientPort=" + ZOOKEEPER_PORT + "' > zookeeper.properties\n";
            command += "echo 'dataDir=/var/lib/zookeeper/data' >> zookeeper.properties\n";
            command += "echo 'dataLogDir=/var/lib/zookeeper/log' >> zookeeper.properties\n";
            command += "zookeeper-server-start zookeeper.properties &\n";
}

And by default Zookeeper is always started (externalZookeeperConnect is not initialized). Also there's no API to distinguish between two modes (with and without Zookeeper) which I guess will be useful for the developers.

[eddumelendez]

Hi @sergey-morenets, thanks for raising the issue! are you interested to work on a PR adding this feature?

[sergey-morenets]

Hi @eddumelendez

Yes, it'd be nice to try that. Although I've never committed to the open-source. So how can I help here? Any instructions?

[eddumelendez]

glad to see you are interested in. I wrote some steps in the past using github cli here. You can do it via github ui too.

[sergey-morenets]

Thank you, @eddumelendez Do you have any suggestions/ideas regarding the design and functionality for this task?

[kiview]

As a first step, I would experiment with using Kafka without Zookeeper and how the container would need to be configured, to be usable in the Testcontainers context. Once you understand how this work, we would probably transition the module to allow 3 different modes:

• External Zookeeper
• Internal Zookeeper
• Kafka as Zookeeper

You can add tests to KafkaContainerTest for the new functionality and get inspiration from the existing tests and the container implementation.

[sergey-morenets]

Hi @kiview

Thank you, I got your idea. Unfortunately I can't check the latest Kafka containers due to this blocker: https://github.com/testcontainers/testcontainers-java/issues/4781

So once it will be resolved I can start working on this ticket.

[pkeugine]

Hi, @kiview Is this issue currently in progress? If not, @ecsimsw and I would like to work on it.

[kiview]

Sure, have a look at it @pkeugine and @ecsimsw @ecsimsw. As outlined in the comments, you might need to have a look at https://github.com/testcontainers/testcontainers-java/issues/4781 as part of it.

[eddumelendez]

pointing to kraft mode (kafka without zookeeper) documentation as a reference. @pkeugine @ecsimsw let us know if you want to discuss something. You can reach out via slack too

[kiview]

Further update, #4781 is not in effect anymore, so this issue has been unblocked.

[pkeugine]

thanks for the update @kiview, @ecsimsw and I read the comments as well. We'll focus on this issue from now on.

[danielpetisme]

Hi,

A made a Kraft POC. https://github.com/testcontainers/testcontainers-java/pull/6129

Feedbacks are welcome!

[eddumelendez]

It's part of version 1.18.0