crowdsecurity
Bug/crowdsec: syslog parser can glue multiple messages on medium or high workload
Describe the bug We deploy crowdsec with syslog datasource and syslog parser in the K8s. Also we monitor in the Grafana incoming packets to the container and acquisitions by pod. On low workload both metrics have close values. But on medium or high workload incoming packets metric exceeds acquisitions one by 1.5-2 times. I assume that the problem is that at the time of reading from the buffer, it contains more than one packet, and message size is much less than MaxMessageLen
To Reproduce Steps to reproduce the behavior:
- Deploy Crowdsec in the K8s with Prometheus and Grafana
- Add one Grafana panel with expression:
max(rate(container_network_receive_packets_total{pod=~"crowdsec-.*"}[1m])) by (pod) - Add another Grafana panel with expression:
sum(rate(cs_syslogsource_hits_total[1m])) by (pod) - Send some logs to the Crowdsec and see the difference
Expected behavior Incoming packets metric and acquisitions metric should show close values
Screenshots
On the last screenshot each packet is shown four times walking through each interface from node to container. But time difference between 2 packets is about 500us. Both packets are coming from same IP and port.
Technical Information (please complete the following information):
- OS CentOS Stream 8
- Crowdsec version 1.3.4
Additional context Bug location: https://github.com/crowdsecurity/crowdsec/blob/1c0fe095768ba01dc3d05db91085d95d4b0d6852/pkg/acquisition/modules/syslog/internal/server/syslogserver.go#L78
Hello @aleksey-novikov,
Could you try to update to 1.4.1 ?
The syslog datasource was improved in this release (better parser, and performance should be better).
Closing due to no response from OP, please reopen if issue persists in 1.4.1 or 1.4.2 (latest)
