Support Kafka as a Sink

[dlvenable]

Is your feature request related to a problem? Please describe.

Some pipeline authors want to send events from Data Prepper into Kafka. This can allow them to use an existing Kafka stream or send data to other destinations via Kafka.

Describe the solution you'd like

Create a Kafka sink to send events to Kafka. It should batch these events when sending to Kafka.

sink:
- kafka:
    bootstrap_servers:
      - localhost:9092
    topic: my-topic

The configuration should also support many of the producer configs. It should certainly support the configurations which the Kafka client will handle completely and do not require anything from Data Prepper beyond passing the configurations.

The Kafka producer client supports sending to multiple topics. If there is value in making topics dynamic, the configuration could support a parameterized topic. For example, topic: my-topic:${/type}.

For connections, this plugin should use the existing mechanisms in Data Prepper for getting SSL certificates, such as from the file, S3, or ACM.

Additional context

https://kafka.apache.org/33/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html

[sharraj]

We should allow DLQ support also here.

[nofar9792]

are you going to add it to your road map?

[zvischutz]

I am starting to work on this one in slow-pace mode . If you know of anyone that started it or wants to participate, please let me know.

[ashoktelukuntla]

The producer should support Plain, SASL_SSL, oAuth and configurable compression types.

log-pipeline:
  source:
    kafka:
      bootstrap_servers:
        - 127.0.0.1:9093
        - 127.0.0.1:9094
        - 127.0.0.1:9095
      auth_type: plaintext
      compression_type: none/gzip   /* optional , #default to none  */
      batch_size: 16384  /* optional , #default 16384  */
      max_request_size: 1MB  /* Optional ,#default 1048576 */
      topic: my-topic-1
      authentication:
          sasl_plaintext:
             username: admin
             password: admin-secret

[tustunkok]

Hi there,

Almost two and a half years since this enhancement have proposed. Are there any plans to implement this feature in the near future? We are constantly checking this feature to convert our Logstash stacks to Data Prepper.

Kind regards,