redpanda-data
Allow multiple docs in stream file
This PR propose a new way to handle stream files.
It is natural to have multiple streams specified in one stream YAML file. For instance, scripts could generate several stream descriptions that may look alike or fall into the same category of functionalities. A straightforward approach by these scripts is to group those stream configurations and bundle them into one YAML file, instead of individual YAML files.
This PR will enable support for multiple documents in a stream file. Stream IDs are assigned in a way such that a suffix of form @<document index> is appended to the major stream ID inferred from the path of a stream file.
There are a few unresolved design details pending discussion. For instance, if a stream is updated with a multi-doc, should benthos yank the the original single stream?
Hey @dingxiangfei2009 very cool. I think the least surprising outcome would likely be removing the single stream in favour of the new identified streams. Needs a bit of thought, maybe an experimental flag in the docs to outline that multidocs are subject to being reworked.
Just in case, I am currently working on putting this behind an experimental flag.
Hey @dingxiangfei2009 are you still interested in this PR? I think realistically before we could consider supporting this we'd need to have a clear answer around linting and config unit tests for these multi document files. It's quite a large undertaking as there's lots of moving parts.