Categorize configuration properties in the output of "kcctl describe connector"

[gunnarmorling]

Connector props should be shown in a grouped way:

• converter properties
• SMTs
• predicates
• topic creation config
• regular connector options

[SiddiqueAhmad]

is there any doc where they are already shown under above categories ? Or we devise the group wise display and later we can link prop to categories

[gunnarmorling]

I was thinking of the following groups to begin with:

• regular connector config properties
• converters and their properties
• SMTs and their properties
• topic creation setttings

[SiddiqueAhmad]

Want to start on this weekend. Should I create one map for group and properties above? Any if already exists in kafka connect libs or it is new concept specific for kcctl

[gunnarmorling]

Should I create one map for group and properties above?

I'm not exactly sure what you mean by this; I was think of something like the following, i.e. group the properties with new lines between the groups (putting comments in there just for sake of discussion, I don't think we necessarily have to put this sort of "label" to the actual output:

Config:
  # Common
  connector.class:  dev.morling.kcetcd.source.EtcdSourceConnector
  tasks.max:        3
  name:             etcd-test
  topics: (only for sinks)

  # Connector properties; i.e. anything not following into the other categories
  clusters:         etcd-a=http://etcd-a-1:2379,http://etcd-a-2:2379,http://etcd-a-3:2379;etcd-b=http://etcd-b-1:2379;etcd-c=http://etcd-c-1:2379
  foo:                 bar

  # SMTs, see https://kafka.apache.org/documentation/#connect_transforms
  transforms=Reroute
  transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter
  transforms.Reroute.topic.regex=(.*)customers_shard(.*)
  transforms.Reroute.topic.replacement=$1customers_all_shards
  transforms.Reroute.key.field.name=shard_id

  # SMT predicates, see https://debezium.io/documentation/reference/transformations/applying-transformations-selectively.html

  predicates=IsOutboxTable
  predicates.IsOutboxTable.type=org.apache.kafka.connect.predicates.TopicNameMatch
  predicates.IsOutboxTable.pattern=outbox.event.*

  # Converters and their properties, see https://kafka.apache.org/documentation/#connect_configuring
  value.converter:  org.apache.kafka.connect.storage.StringConverter
  key.converter:    org.apache.kafka.connect.storage.StringConverter

  # Topic Creation settings (see https://debezium.io/documentation/reference/configuration/topic-auto-create-config.html)
  topic.creation.inventory.include:                     dbserver1\\.inventory\\.*
  topic.creation.inventory.partitions:                 20
  topic.creation.inventory.cleanup.policy:         compact
  topic.creation.inventory.delete.retention.ms: 7776000000

Any if already exists in kafka connect libs or it is new concept specific for kcctl

The different kinds and purposes of options exists as a concept in Kafka Connect. To properly organize all the properties of a connector into these groups, you'd have to examine the different key prefixes likes topic.creation., predicates, etc.

[sudoHackIn]

Hi, @gunnarmorling. Is it still actual? Am i correctly understand, that for this issue, we should create EtcdSourceConnector config file example with all existing properties, grouped by?

[gunnarmorling]

Hey @sudoHackIn, yes, this one is still up for grabs, if you'd like to give it a shot.

Am i correctly understand, that for this issue, we should create EtcdSourceConnector config file example with all existing properties, grouped by?

So I was showing the EtcdSourceConnector configuration mainly as an example for which kinds of properties exist in general, and how the output could look like (I often use the etcd connector for demos, mainly because it supports multiple source tasks; there's a Compose file for starting up a few etcd clusters here). So you could use an EtcdSourceConnector configuration file for developing and testing this. In terms of automated tests, were're not quite there yet, see the pending PR for it.

[gunnarmorling]

Groups would be these for a starter:

• regular connector config properties
• converters (and header converters) and their properties
• SMTs and their properties
• topic creation settings

As a follow-up, we may also think about grouping connector properties themselves (as Kafka Connect allows to organize them in field groups), but I think this could be handled separately.

[sudoHackIn]

Hey @sudoHackIn, yes, this one is still up for grabs, if you'd like to give it a shot.

Am i correctly understand, that for this issue, we should create EtcdSourceConnector config file example with all existing properties, grouped by?

So I was showing the EtcdSourceConnector configuration mainly as an example for which kinds of properties exist in general, and how the output could look like (I often use the etcd connector for demos, mainly because it supports multiple source tasks; there's a Compose file for starting up a few etcd clusters here). So you could use an EtcdSourceConnector configuration file for developing and testing this. In terms of automated tests, were're not quite there yet, see the pending PR for it.

Sorry, i am also very new to kafka connect, so still cannot fully understand this issue :( So the task is to document and write down(grouped by) available properties for better and easy connector "structure" understanding? Without parsing and config validation code changes? Because, as i understand in ApplyCommand https://github.com/kcctl/kcctl/blob/c0c6e473264da93fddd491fd9e28fb85fb5b9a22/src/main/java/org/kcctl/command/ApplyCommand.java#L109 we mainly pass configuration file "as is" to kafka endpoint, and do not need to create special class structure for properties in code.

[gunnarmorling]

No worries, I see how the issue description is lacking some context.

So this is about improving the output of the kcctl describe connector <NAME> command (as you say, when creating a connector via kcctl apply, the configuration must be passed as-is). Currently, all the properties will be shown in one continuous list of key/value pairs, as seen in the Config section here. I think this could improved by grouping the properties (using empty lines between the groups), as shown here.

In the linked video, the shown connector config only contains properties of the Debezium connector, i.e. there would be no change (for now) how they are displayed, as they all are part of one group (the connector's specific poperties). But Kafka Connect defines a range of other properties for configuring converters, SMTs, SMT predicates, and topic creation settings. If any of those are present for a given connector, they should be shown in separate groups then, as in the example given above.

Does that make sense?

[sudoHackIn]

No worries, I see how the issue description is lacking some context.

So this is about improving the output of the kcctl describe connector <NAME> command (as you say, when creating a connector via kcctl apply, the configuration must be passed as-is). Currently, all the properties will be shown in one continuous list of key/value pairs, as seen in the Config section here. I think this could improved by grouping the properties (using empty lines between the groups), as shown here.

In the linked video, the shown connector config only contains properties of the Debezium connector, i.e. there would be no change (for now) how they are displayed, as they all are part of one group (the connector's specific poperties). But Kafka Connect defines a range of other properties for configuring converters, SMTs, SMT predicates, and topic creation settings. If any of those are present for a given connector, they should be shown in separate groups then, as in the example given above.

Does that make sense?

Yes, clearly, thanks) will start working on the issue tomorrow