client has run out of available brokers to talk to: EOF

[WoodenMaiden]

Describe the bug When creating a knative broker using a kafka cluster created by Bitnami's kafka Helm chart, it shows this error:

yann@xaku$ k get brokers
NAME      URL   AGE   READY   REASON
default         17h   False   cannot obtain Kafka cluster admin, kafka: client has run out of available brokers to talk to: EOF

Expected behavior Deploy a working kafka event source following the documentation at https://knative.dev/docs/eventing/brokers/broker-types/kafka-broker/#installation

To Reproduce

1. install the kafka cluster

helm install kafka-eventing oci://registry-1.docker.io/bitnamicharts/kafka \
  --version 31.5.0 \
  --namespace <kafka_namespace> \
  --create-namespace \
  --atomic \
  --cleanup-on-fail \
  --wait \
  --set broker.replicaCount=3 \
  --set controller.replicaCount=1

2. install knative and the kafka broker layer

3. Create kafka-broker-config configmap

apiVersion: v1
data:
  bootstrap.servers: kafka-eventing.kafka:9092
  default.topic.config.retention.ms: "3600"
  default.topic.partitions: "10"
  default.topic.replication.factor: "1"
kind: ConfigMap
metadata:
  name: kafka-broker-config
  namespace: knative-eventing

4. Create config-br-defaults

apiVersion: v1
data:
  default-br-config: |-
    clusterDefault:
      brokerClass: Kafka
      apiVersion: v1
      kind: ConfigMap
      name: kafka-broker-config
      namespace: knative-eventing
kind: ConfigMap

5. Create the Broker

apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
  annotations:
    eventing.knative.dev/broker.class: Kafka
  name: default
  namespace: default

Knative release version v1.17.3

[matzew]

IS there a service for the kafka broker? I do not know the helm chart.

Like, in my case, when I fetch the services in the kafka namespace:

k get services -n kafka
NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                   AGE
my-cluster-kafka-bootstrap    ClusterIP   10.96.238.135   <none>        9091/TCP,9092/TCP,9093/TCP,9094/TCP,9095/TCP,9096/TCP                     17s
my-cluster-kafka-brokers      ClusterIP   None            <none>        9090/TCP,9091/TCP,8443/TCP,9092/TCP,9093/TCP,9094/TCP,9095/TCP,9096/TCP   17s
my-cluster-zookeeper-client   ClusterIP   10.96.93.209    <none>        2181/TCP                                                                  56s
my-cluster-zookeeper-nodes    ClusterIP   None            <none>        2181/TCP,2888/TCP,3888/TCP                                                56s

I see the my-cluster-kafka-bootstrap service in there, with port 9092 (and others). so my ConfigMap looks like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-broker
  namespace: sacura
data:
  default.topic.partitions: "10"
  default.topic.replication.factor: "3"
  bootstrap.servers: "my-cluster-kafka-bootstrap.kafka:9092"

FWIW, we use Strimzi Operator in our e2e setup: https://github.com/knative-extensions/eventing-kafka-broker/tree/main/test/kafka

Perhaps make it a string:

bootstrap.servers: "kafka-eventing.kafka:9092"

[matzew]

@WoodenMaiden hey 👋 Wondering if you've been able to figure out some more on this issue. See also my above comment

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

[joint-song]

I also have this error after creating a broker with an old version kafka cluster(0.11 broker api version). Changing to a kafka cluster with 3.4 resolved this problem. Documenting the supported kafka version will be better?

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.