Setting advertised listeners is not propagated to broker

[berndfo]

Affected version

docker.stackable.tech/stackable/kafka-operator:23.1.0

Current and expected behavior

Requirement: Giving access to Kafka brokers from within a k8s cluster.

Currently, only the external cluster IP is per default advertised by brokers. For my particular setup, this IP:port address isn't and shouldn't be accessible, not from inside or outside of the cluster.

When trying to set the advertised.listeners property of Kafka through KafkaCluster, it is not picked up by the broker configuration, see configuration below. This results in Kafka being effectively not usable in this particular scenario.

  brokers:
    roleGroups:
      default1:
        replicas: 1    
      rg1:
        replicas: 1
        configOverrides:
          server.properties:
            advertised.listeners: CLIENT://stack-kafka-broker-rg1-0.mynamespace:9092

kcat shows that brokers still advertise the original external ip regardless of the override.

the brokers configmap contains the override:

5 apiVersion: v1
6 data:
7   log4j.properties: ""
8   server.properties: |
9     advertised.listeners=CLIENT\://stack-kafka-hbdedpmdsf-broker-rg1-0.mynamespace\:9092

yet, the broker itself still does it's own override using the node port (CLIENT://$NODE:$(cat /stackable/tmp/kafka_nodeport)), not preferring the intended override:

  bin/kafka-server-start.sh /stackable/config/server.properties --override "zookeeper.connect=$ZOOKEEPER" --override "listeners=CLIENT://0.0.0.0:9092,INTERNAL://0.0.0.0:19093" --override "advertised.listeners=CLIENT://$NODE:$(cat /stackable/tmp/kafka_nodeport),INTERNAL://$POD_NAME.stack-kafka-hbdedpmdsf-broker-rg1.mynamespace.svc.cluster.local:19093" --override "listener.security.protocol.map=CLIENT:PLAINTEXT,INTERNAL:SSL"

Possible solution

No response

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

None

[terrancesnyder]

Also have a similar issue, config options dont get set. They do find themselves below;

bin/kafka-server-start.sh /stackable/config/server.properties

But are overridden in

https://github.com/stackabletech/kafka-operator/blob/db3c918a5ed53ae5a80287bb947fca42304b4249/rust/crd/src/listener.rs#L58

Where its always using these settings.