using the concat plugin with CRI logs ends up deleting the stream key from the record

[mmerickel]

What is the purpose of dropping the stream key when combining records? It seems intentional as part of comments made in https://github.com/fluent-plugins-nursery/fluent-plugin-concat/commit/cb96cc7fb98ff2398c10b599a4efa5a82707cb64 but I can't see why.

Further, the example linked and used in tests doesn't make any sense that a partial message would not have the same stream value across the partials. You should think that each stream would be concatenated individually and then the stream tag preserved.

This issue is very similar to https://github.com/fluent/fluent-bit/issues/4387 where they are also parsing it incorrectly, It would be nice if fluentd-concat would get this right. You can see the example in this linked issue below and will be parsed incorrectly by the concat plugin.

2021-11-18T14:47:34.730832241+00:00 stdout F some short log message A
2021-11-18T14:47:34.730832241+00:00 stdout P some longer log message B part one
2021-11-18T14:47:34.730908438+00:00 stdout F end of the longer message B
2021-11-18T14:47:34.730908438+00:00 stdout P another longer log message  C part one
2021-11-18T14:47:34.730939216+00:00 stdout P message C continued in part two
2021-11-18T14:47:34.735454549+00:00 stderr F some log message D in stderr part one
2021-11-18T14:47:34.735482854+00:00 stdout F message C still continued after a line from a different stream
2021-11-18T14:47:34.735482854+00:00 stderr F and the end of the stderr message D

[ardole]

+1

same question for me !

my use case is on CRI logs; i wanted to use rewrite_tag_filter based on stream key, in order to send stderr/stdout to different target...

so i got a problem when stream key is removed by concat filter.

i know that i can use routing/splitting by labels to fix my issue, but still wondering why stream key is removed without any way to keep it.