examples icon indicating copy to clipboard operation
examples copied to clipboard

Enhance the clickstream ksqlDB demo to create connectors using ksqlDB

Open big-andy-coates opened this issue 5 years ago • 3 comments

You can now create connectors within ksqlDB. We're already using SOURCE connectors within the demo. We should update the demo to:

  • Not run connect as a separate container, but run within ksqlDB
  • Use the CREATE SINK CONNECTOR syntax in ksqlDB to create the connectors to sink to ES, and remove ksql-connect-es-grafana.sh.

The whole create source connectors, execute sql statements, create sink connectors, could be in a single SQL script....

big-andy-coates avatar Jun 11 '20 10:06 big-andy-coates

Not run connect as a separate container, but run within ksqlDB

In 5.5.0-post, connect is not a separate container, it is already run within ksqlDB.

ybyzek avatar Jun 11 '20 11:06 ybyzek

Use the CREATE SINK CONNECTOR syntax in ksqlDB to create the connectors to sink to ES, and remove ksql-connect-es-grafana.sh.

So that leaves the question about the second bullet point, whether the Elasticsearch sink connectors should also use the CLI CREATE SINK CONNECTOR. The demo intentionally does not do that in order to show users how to use the REST API to the connect cluster.

ybyzek avatar Jun 11 '20 11:06 ybyzek

I've always viewed the clickstream demo as a ksqlDB demo, and so I'm be leaning towards using ksqlDB features wherever possible. Maybe you view it more as a CP demo, and in which case I can see why this would be useful.

If you feel it's more the latter, then feel free to close this :D

big-andy-coates avatar Jun 11 '20 11:06 big-andy-coates