Use existing topic config during stream PUT if topic already exits but is not registered

[cstrombe]

Desired Behavior

We need to programmatically register streams via the streams PUT call. This is failing with the error {"code":400,"message":"Error: Input configs={} and actual configs={min.insync.replicas=2} are not same for topic=<stream name>" when we issue a PUT to the endpoint v0/streams/<stream name>

Desired behavior:

• if no topic exists, create the topic with the supplied config
• If the topic already exists and is not registered with the stream registry, register the stream with the topic's current config
• If the topic already exists and is registered with the stream registry, then it's a no-op

Benefits

Please list the benefits of updating the stream-registry project to have the new behavior. E.g.:

Enables programmatic stream registration for different use cases.

[neoword]

@cstrombe thanks ... will have the team take a look at this.

[pthatchinamoorthy]

I believe, @cstrombe encountered this error for the scenario 2 (topic already exists in kafka, but not in streamregistry) In that scenario, throwing an error is the better behavior than

If the topic already exists and is not registered with the stream registry, register the stream with the topic's current config

Lets say, some one triggers their StreamRegistration call using a Automation Script or Terraform (checked in Git) with config={retention.ms=1year} for a Actual kafka Topic with config={retention=7days} then, if we register the stream with topic's current config, then the team would always assume that thier topic has the retention of 1 year which is not wrong.

[cstrombe]

It kind of seems like if the topic exists already, then it's safe to assume that the config it has is the desired config. One idea would be to include a parameter in the endpoint payload to make the choice explicit if topic already exists--"use the incoming config" or "keep existing config". And/or issue a warning in logs (and in return from endpoint call) that "existing topic config retained". But maybe automatic stream registration is just not possible in all cases...anyway, just some ideas. Thanks for listening!

[neoword]

@cstrombe would a "-f" or "force==true" flag make sense ? Or maybe called it "sync=true" ? E.g. Make the stream metadata match the backend if it exists, failing otherwise ?