Beat via Kafka

[xuziheng1002]

we use graylog and sidecar to collector application logs,but graylog-server input has been the bottle-neck, so we think it will be better if let Beat input get messages from kafka. Is there any schedule to develop this feature now?

[florianpopp]

We do not have anything scheduled yet, but this is a valid request.

[BenoitPoulet]

is it on the roadmap ?

You are pushing Filebeat to ingest logs, but we are unable to connect it to a queue.

[BenoitPoulet]

any news about this ?

[rkmbaxed]

News would be great, we are also waitung for this feature.

[waab76]

Is there a reason that the existing Kafka inputs don't work for this case? What additional functionality are we hoping to get from specifically building a Kafka input for Beats?

[bernd]

@waab76 We can reuse the Kafka transport infrastructure but need a new Kafka-based input with a Beats codec. We have to find out how the Beats payload looks like. If it's the same binary payload that is used in the lumberjack protocol, we can probably reuse the Beats input's codec.

[waab76]

Tentatively targeting this for 6.1

[github-actions[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[bernd]

@waab76 I reopened this issue because you mentioned that you are targeting it.

[ryan-carroll-graylog]

New input added in https://github.com/Graylog2/graylog2-server/pull/20067