fluent-plugin-concat log with lengthy exception is getting dropped

Problem

Trying to concatenate Multi line records into single log record Working well for all kinds of exceptions/startup logs Lengthy exceptions are getting dropped. ...

Steps to replicate

Please tryout below config and seeing lengthy lines/exceptions are getting dropped. Would like to know if we can limit lines or increase if its related to Buffer limitation

Provide example config and message

2020-04-03T02:57:17.605Z ERROR

[Server] FATAL error: com.vmware.vdi.pop.configuration.ConfigurationException$IOConfigurationException: Failed to read configuration com.vmware.vdi.pop.configuration.ConfigurationException$IOConfigurationException: Failed to read configuration at com.vmware.vdi.pop.configuration.ConfigurationManager.read(ConfigurationManager.java:74) at com.vmware.vdi.pop.configuration.ConfigUtils.readConfig(ConfigUtils.java:88) at com.vmware.vdi.hydra.connector.server.Server.main(Server.java:287) Caused by: java.io.FileNotFoundException: ./tls/connector-api.config (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:916) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3070) at com.vmware.vdi.pop.configuration.ConfigurationManager.read(ConfigurationManager.java:68) ... 2 common frames omitted Native Memory Tracking: Total: reserved=2315290KB, committed=68470KB

            Java Heap (reserved=983040KB, committed=32768KB)
                      (mmap: reserved=983040KB, committed=32768KB)

                Class (reserved=1063273KB, committed=16489KB)
                      (classes #2827)
                      (malloc=361KB #1208)
                      (mmap: reserved=1062912KB, committed=16128KB)

               Thread (reserved=5718KB, committed=5718KB)
                      (thread #10)
                      (stack: reserved=5672KB, committed=5672KB)
                      (malloc=33KB #52)
                      (arena=13KB #19)

                 Code (reserved=250014KB, committed=3342KB)
                      (malloc=414KB #1119)
                      (mmap: reserved=249600KB, committed=2928KB)

                   GC (reserved=3215KB, committed=123KB)
                      (malloc=7KB #84)
                      (mmap: reserved=3208KB, committed=116KB)

             Compiler (reserved=138KB, committed=138KB)
                      (malloc=7KB #90)
                      (arena=131KB #5)

             Internal (reserved=506KB, committed=506KB)
                      (malloc=474KB #3772)
                      (mmap: reserved=32KB, committed=32KB)

               Symbol (reserved=4910KB, committed=4910KB)
                      (malloc=3815KB #30989)
                      (arena=1095KB #1)

Native Memory Tracking (reserved=589KB, committed=589KB) (malloc=3KB #35) (tracking overhead=586KB)

          Arena Chunk (reserved=3887KB, committed=3887KB)
                      (malloc=3887KB)_

Expected Behavior

concat plugin should be able concat above multiline exception/message into single log record. OR Need a way to truncate instead of dropping entire message. ...

Your environment

Concat Plugin Config <filter **> @type concat key log multiline_start_regexp /^\d{4}[-/]\d{1,2}[-/]\d{1,2}/ separator "" flush_interval 5

OS version Linux dev1-westus2-jumpbox 5.0.0-1028-azure #30~18.04.1-Ubuntu SMP Fri Dec 6 11:47:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
paste result of fluentd --version or td-agent --version
plugin version
- paste boot log of fluentd or td-agent
- paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch

Apr 03 '20 03:04 NareshAnaparthi

Any updates on this? Having also issues with concatenation of multineline logs and multiline_start_regexp.

Sep 02 '20 07:09 kaisermario

Hmm, I cannot reproduce the issue.

Sep 04 '20 09:09 okkez