influxdata
[[output.nats]] Bytes limits on output size
Use Case
In NATS, messages have a max size (1Mb by default).
As telegraf limits the size of the batch in number of messages (instead of bytes), it's not very useful in the case when one message can contain multiple lines of lineprotocol output, making messages' actual length very wide.
Expected behavior
It is possible to limit the messages sent to NATS to avoid triggering the message limit size.
This feature is already implemented on other output plugins such as cloud_pubsub, event_hubs, file, graylog and influxdb (thanks @Hipska for the research).
Actual behavior
Given a big enough metric_batch_size value, NATS max message size error is triggered when the serialized message exceeds the max message size (1Mb by default).
Additional info
No response
Is this option not offered by the Jetstream max_bytes option provided by this plugin, or am I misunderstanding something?
My understanding is that max_bytes is the "Maximum number of bytes stored in the stream. Adheres to Discard Policy, removing oldest or refusing new messages if the Stream exceeds this size." (from https://docs.nats.io/nats-concepts/jetstream/streams) and that would be a setting on how the stream is created on NATS.
What the issue is about is the size of each individual message sent to NATS (being sent to a JetStream subject or a non jetstream one).
There is more information on slack, on this thread. Here I copy some relevant messages:
A size limit for the output is certainly not a Telegraf-wide feature as some plugins will not be able to support arbitrary size limitation (e.g. when sending out fixed binary formats). Furthermore, as [@Hipska](https://influxcommunity.slack.com/team/UQD4RV7RU) already said, Telegraf itself (i.e. the buffer) cannot know the size of the serialized metric and thus cannot limit the size.
However, a size limitation can be useful on a plugin basis e.g. if there are limitations like for Nats...
This being said, a size limited serializer is already available in [/plugins/common/ratelimiter/serializers.go](https://github.com/influxdata/telegraf/blob/master/plugins/common/ratelimiter/serializers.go), so somebody needs to wire this up into the nats output plugin. The idea is to only consume so many metrics that we stay below the size-limit. However, currently the implementation above only supports serializers that can iteratively add data e.g. line-protocol but not serializers that need to see the complete batch of data e.g. XML or JSON.
A size-limited serializer supporting batch-based serializers is doable by bisecting the batch of metrics to find the optimal batch but it is a) not easy and b) potentially computationally expensive as you are potentially serializing the same batch multiple times... (edited)
It was also noted that:
Aha, so the NATS server communicates this limitation to the client when connecting, then it doesn’t even need to be a seperate telegraf configuration. Just take that value in consideration when creating the messages..
@inakimalerba is this not useful to you https://github.com/influxdata/telegraf/blob/master/plugins/outputs/nats/nats.go#L62
@neelayu AFAIK the MaxMsgSize is a configuration used when creating a new JetStream. This can be a limit lower than the global NATS max message size (usually 1MB). The global limit is sent by the nats server when connecting. I guess that if this feature is implemented it should actually used the smallest of the two, the global limit and the particular limit of the JetStream being published to. But just setting value will not limit the messages telegraf tries to send.