Kafka out failure in name resolution

[Lubard]

Hi, we are using this docker image version 1.2.1, and trying to connect send out to Kafka. And we host the kafka brokers on our own servers. Here is the log we got:

[2020/02/25 10:24:36] [ info] [storage] initializing...

[2020/02/25 10:24:36] [ info] [storage] in-memory

[2020/02/25 10:24:36] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128

[2020/02/25 10:24:36] [ info] [engine] started (pid=1)

[2020/02/25 10:24:36] [ info] [out_kafka] brokers='10.37.XXX.XX:9092,10.37.XXX.YY:9092' topics='cpu_check'

[2020/02/25 10:24:36] [ info] [sp] stream processor started

[2020/02/25 10:24:56] [error] [out_kafka] fluent-bit#producer-1: [thrd:clrv0000160257.ic.ing.net:9092/0]: clrv0000XXX.net:9092/0: Failed to resolve 'clrv0000XXX.net:9092': Temporary failure in name resolution (after 3461538871ms in state INIT)

[2020/02/25 10:24:56] [error] [out_kafka] fluent-bit#producer-1: [thrd:clrv0000160257.ic.ing.net:9092/0]: clrv0000XXX.net:9092/0: Failed to resolve 'clrv0000XXX.net:9092': Temporary failure in name resolution (after 3461538871ms in state INIT)

[2020/02/25 10:24:56] [error] [out_kafka] fluent-bit#producer-1: [thrd:clrv0000160263.ic.ing.net:9092/1]: clrv0000YYY.net:9092/1: Failed to resolve 'clrv0000YYY.net:9092': Temporary failure in name resolution (after 3461538871ms in state INIT)

[2020/02/25 10:24:56] [error] [out_kafka] fluent-bit#producer-1: [thrd:clrv0000160263.ic.ing.net:9092/1]: clrv0000YYY.net:9092/1: Failed to resolve 'clrv0000YYY.net:9092': Temporary failure in name resolution (after 3461538871ms in state INIT)

clrv0000XXX.net is the machine name of the first broker 10.37.XXX.XX clrv0000YYY.net is the machine name of the second broker 10.37.XXX.YY

It seems the connection was successful as Fluent-bit has created the intended topic on Kafka side, but cannot push data through.

After few minutes, we received this:

[2020/02/25 10:29:45] [ warn] [out_kafka] message delivery failed: Local: Message timed out
[2020/02/25 10:29:45] [ warn] [out_kafka] message delivery failed: Local: Message timed out
[2020/02/25 10:29:45] [ warn] [out_kafka] message delivery failed: Local: Message timed out

Any ideas why? Our Kafka servers are running just fine with other applications.

---

Update: Update the advertised.listeners to brokers' own IP address fixed this issue. But it does not optimised as all machines are running under the same network...