add new type of client - for Kafka protocol

[derberg]

Let's explore and try adding a client for a new protocol - kafka

so bacially create a new folder in https://github.com/asyncapi/generator/tree/master/packages/templates/clients

this work will trigger new discussions on:

• shoudl we actually have common tests in https://github.com/asyncapi/generator/tree/master/packages/templates/clients/websocket/test or rather on clients dir level
• can we reuse components between templates supporting different protocols but the same languages? so can we reuse (maybe after some refactoring) components from existing javascript template for websocket in the new kafka template

probably more questions pop up in our heads.

[nightknighto]

I'd vote for using a modern Kafka library: https://www.npmjs.com/package/kafkajs instead of Hermes for JavaScript.

[derberg]

yeah 100% not hermes. The one you mentioned seems to be the best match

[ssala034]

@nightknighto would you like to work on this together?

[nightknighto]

@nightknighto would you like to work on this together?

Sure. Making a reusable and extendable SDK on top of Kafka js could be challenging.

[nightknighto]

Let's start by getting some example documents for Kafka. @derberg Do you have a few documents to use as a reference? Preferably one simple and one complex.

[nightknighto]

asyncapi: 3.0.0
info:
  title: Streetlights Kafka API
  version: 1.0.0
  description: |-
    The Smartylighting Streetlights API allows you to remotely manage the city
    lights.
servers:
  development:
    host: 'test.mykafkacluster.org:18092'
    protocol: kafka
    description: Test broker
channels:
  lightingMeasured:
    address: '{streetlightId}.lighting.measured'
    messages:
      lightMeasured:
        $ref: '#/components/messages/lightMeasured'
    description: The topic on which measured values may be produced and consumed.
    parameters:
      streetlightId:
        $ref: '#/components/parameters/streetlightId'
  lightTurnOn:
    address: '{streetlightId}.turn.on'
    messages:
      turnOn:
        $ref: '#/components/messages/turnOnOff'
    parameters:
      streetlightId:
        $ref: '#/components/parameters/streetlightId'
  lightTurnOff:
    address: '{streetlightId}.turn.off'
    messages:
      turnOff:
        $ref: '#/components/messages/turnOnOff'
    parameters:
      streetlightId:
        $ref: '#/components/parameters/streetlightId'

operations:
  receiveLightMeasurement:
    action: receive
    channel:
      $ref: '#/channels/lightingMeasured'
    summary: >-
      Inform about environmental lighting conditions of a particular
      streetlight.
    traits:
      - $ref: '#/components/operationTraits/kafka'
    messages:
      - $ref: '#/channels/lightingMeasured/messages/lightMeasured'
  turnOn:
    action: send
    channel:
      $ref: '#/channels/lightTurnOn'
    traits:
      - $ref: '#/components/operationTraits/kafka'
    messages:
      - $ref: '#/channels/lightTurnOn/messages/turnOn'
  turnOff:
    action: send
    channel:
      $ref: '#/channels/lightTurnOff'
    traits:
      - $ref: '#/components/operationTraits/kafka'
    messages:
      - $ref: '#/channels/lightTurnOff/messages/turnOff'

components:
  messages:
    lightMeasured:
      name: lightMeasured
      title: Light measured
      summary: >-
        Inform about environmental lighting conditions of a particular
        streetlight.
      contentType: application/json
      traits:
        - $ref: '#/components/messageTraits/commonHeaders'
      payload:
        $ref: '#/components/schemas/lightMeasuredPayload'
    turnOnOff:
      name: turnOnOff
      title: Turn on/off
      summary: Command a particular streetlight to turn the lights on or off.
      traits:
        - $ref: '#/components/messageTraits/commonHeaders'
      payload:
        $ref: '#/components/schemas/turnOnOffPayload'

  schemas:
    lightMeasuredPayload:
      type: object
      properties:
        lumens:
          type: integer
          minimum: 0
          description: Light intensity measured in lumens.
        sentAt:
          $ref: '#/components/schemas/sentAt'
    turnOnOffPayload:
      type: object
      properties:
        command:
          type: string
          enum:
            - 'on'
            - 'off'
          description: Whether to turn on or off the light.
        sentAt:
          $ref: '#/components/schemas/sentAt'
    sentAt:
      type: string
      format: date-time
      description: Date and time when the message was sent.
  parameters:
    streetlightId:
      description: The ID of the streetlight.
  messageTraits:
    commonHeaders:
      headers:
        type: object
        properties:
          my-app-header:
            type: integer
            minimum: 0
            maximum: 100
  operationTraits:
    kafka:
      bindings:
        kafka:
          clientId:
            type: string
            enum:
              - my-app-id

I have taken the kafka example on async api and cut some parts from it to get this. Would this be a good start for the document to develop on, @derberg ?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity :sleeping:

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience :heart:

[payal723]

Hi @derberg ,

I see this Kafka client addition is highly desired! I am a JavaScript/Node.js developer with recent experience in migrating Generator templates to V3.

Since this issue is open, I would like to take ownership of the JavaScript/Node.js client implementation using kafkajs.

My plan is:

Create the base directory (clients/kafka) and structure.

Implement the V3 Parser logic to handle Kafka send/receive operations.

Investigate code reusability between the existing Websocket/JS template and the new Kafka template.

I will start the implementation immediately and post updates here. Please let me know if this approach works for you!

Thanks, Payal

[derberg]

@payal723 let's get https://github.com/asyncapi/generator/pull/1749 done first

regarding Kafka, this is bigger topic, there is one client already for Kafka using Quarkus