Apache Kafka Integration

[Fadavvi]

Is your feature request related to a problem? Please describe. No

Describe the solution you'd like Use Apache Kafka Queue(s) as a source of logs. also Kafka can used for sharing xbits & etc cross the network.

Describe alternatives you've considered For second part of my request (sharing xbits), Sagan has Redis support. (but redis is single queue)

Additional context Please check blow link: https://kafka.apache.org/intro#kafka_mq

[beave]

I have a long flight coming up. I'll make sure I read more about Kafka as I've never used it before.

[candlerb]

Make sure you look at Apache Pulsar too: it fits a wider range of use cases and is easier to scale and manage.

Some comparisons:

• https://kafkaesque.io/7-reasons-we-choose-apache-pulsar-over-apache-kafka/
• https://medium.com/@yuvarajl/why-nutanix-beam-went-ahead-with-apache-pulsar-instead-of-apache-kafka-1415f592dbbb
• https://www.jesse-anderson.com/2019/05/reducing-operational-overhead-with-pulsar-functions/