nats_consumer doesn't support specifying the stream for jetstream_subjects

[chris13524]

Use Case

nats_consumer only supports looking up streams by subject (https://github.com/influxdata/telegraf/issues/11046#issuecomment-1165939774), however this approach only works for streams that have subjects specified. Mirrored streams cannot be looked up by subject and must specify the stream name.

To work with mirrored streams, the stream name must be used instead of subjects. Subjects may still be used for filtering (filterSubjects in consumer config)

Expected behavior

Can specify stream to create consumer for

Actual behavior

Can only create consumers given subjects

Additional info

No response

[sspaink]

So looking at the documentation for the package we use for this plugin (https://pkg.go.dev/github.com/nats-io/nats.go#section-readme) it isn't obvious how you could use a stream name instead of a subject. Have you tried adding the stream name to the subjects configuration to see if it would work?

[chris13524]

You specify the stream name using the Bind function: https://pkg.go.dev/github.com/nats-io/nats.go#BindStream

See docs here: https://pkg.go.dev/github.com/nats-io/nats.go#JetStream

If no stream name is specified, the library will attempt to figure out which stream the subscription is for.

If no stream name is specified, the client will attempt lookup the stream using the subject specified. This is the same behavior across NATS clients, so I'm confident that the current design does not work for streams that cannot be looked up using subjects.

[sspaink]

@chris13524 good find! So BindStream looks like an option that gets passed to: https://github.com/influxdata/telegraf/blob/master/plugins/inputs/nats_consumer/nats_consumer.go#L158 ? I assume a new configuration option needs to be added to provide the list of stream names.

Would you be able to perhaps open a pull request with this change?

[chris13524]

Yeah some new sort of configuration would be needed. Would a separate configuration be desired, or a change of the existing configuration design? I'm guessing there is a need for backwards-compatibility, but not sure the ergonomics of multiple configs.

We are considering contributing a PR

[dwrightpulse]

Wondering if any progress has been made on this issue...? I have stumbled across the same constraint today (trying to use Telegraf's nats_consumer input to consume messages from a mirrored stream)...

[dwrightpulse]

If anyone else is struggling with this I was able to configured NATS to republish the messages from a mirrored stream to a new local stream (using the NATS CLI command "nats stream add --republish-source= --republish-destination="). Not sure the syntax of the command is 100% right but it worked for me and then gave me a new local stream where Jetstream was aware of the subject names contained in the stream. This allowed the nats_consumer input to set itself up as a consumer in the correct stream.