fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard
verified publisher icon fluent

filter_throttle_tag: initial commit of msg tag based throttle filter

Open novegit opened this issue 3 years ago • 1 comments

This PR introduces filter_throttle_tag plugin, which is derived from filter_throttle/filter_throttle_size, but much more suitable for kubernetes environments. My company runs 'namespace as a service' on openshift clusters for different customers, and we are often in the situation, that customers start pods with irresponsible high log output, causing sometimes missing logs from other pods on the same node, due to overload. In multiuser kubernetes clusters, granted resources should have the capability to limit them. This plugin limits the logmessages per container and increases the reliability of service. The plugin (on kubernetes) is meant to be used behind the tail plugin, so the overloads can be dropped in early stage of the logging pipeline.

Example configuration:

[FILTER]
    name throttle_tag
    Match       tail*
    Rate        10.0
    Global_rate 50.0
    Window      12
    Interval    5s
    Startup_wait 10s
    Print_Status True

Gobal_Rate: if the msg rate for a tag is reached and a higher global_rate is configured, the msg will pass the throttle, until the global_rate isn't reached. So a higher msg rate for tags can be granted, when enough capacity on a node is free.

Startup_wait: after a fluent-bit downtime on a kubernetes cluster, a high amount of logs from many container logs files must be read. In this situation, we want to bypass the throttle filter for a startup timespan, so no logmessages will be dropped by the plugin.

Example configuration:

[SERVICE]
    flush        2
    daemon       Off
    log_level   info
    http_server  On
    http_listen  0.0.0.0
    http_port    2020
    storage.metrics on
[INPUT]
    name tail
    tag  tail*
    Read_from_Head true
    Refresh_Interval 1
    path /var/tmp/input*
[FILTER]
    name throttle_tag
    Match       tail*
    Rate          10.0
    Global_rate 50.0
    Window      6
    Interval    5s
    Startup_wait 10s
    Print_Status True
[OUTPUT]
    name null

Example output: producing logs via manual commands:

for i in $(seq 1 50); do echo "$i" >> input1 ; echo "$i" >> input2; echo "$i" >> input3 ; done
for i in $(seq 1 50); do echo "$i" >> input1 ; echo "$i" >> input2; echo "$i" >> input3 ; done
for i in $(seq 1 50); do echo "$i" >> input1 ; echo "$i" >> input2; echo "$i" >> input3 ; done
for i in $(seq 1 50); do echo "$i" >> input1 ; echo "$i" >> input2; echo "$i" >> input3 ; done

/fluent-bit/bin/fluent-bit -c /var/tmp/f.conf
Fluent Bit v2.0.6
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/11/28 21:02:05] [ info] [fluent bit] version=2.0.6, commit=3f6d9d7e4d, pid=44430
[2022/11/28 21:02:05] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2022/11/28 21:02:05] [ info] [cmetrics] version=0.5.7
[2022/11/28 21:02:05] [ info] [ctraces ] version=0.2.5
[2022/11/28 21:02:05] [ info] [input:tail:tail.0] initializing
[2022/11/28 21:02:05] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2022/11/28 21:02:05] [ info] [output:null:null.0] worker #0 started
[2022/11/28 21:02:05] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2022/11/28 21:02:05] [ info] [sp] stream processor started
[2022/11/28 21:02:21] [ info] [input:tail:tail.0] inotify_fs_add(): inode=942024 watch_fd=1 name=/var/tmp/input1
[2022/11/28 21:02:21] [ info] [input:tail:tail.0] inotify_fs_add(): inode=942026 watch_fd=2 name=/var/tmp/input2
[2022/11/28 21:02:21] [ info] [input:tail:tail.0] inotify_fs_add(): inode=942087 watch_fd=3 name=/var/tmp/input3
[2022/11/28 21:02:25] [ info] [filter:throttle_tag:throttle_tag.0] Tag: __global_window__, last_msg_time: 1669669340, Total: 150, Dropped: 0, Rate: 25.00, Max_rate: 10.00
[2022/11/28 21:02:25] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input1, last_msg_time: 1669669341, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:02:25] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input2, last_msg_time: 1669669341, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:02:25] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input3, last_msg_time: 1669669341, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:02:30] [ info] [filter:throttle_tag:throttle_tag.0] Tag: __global_window__, last_msg_time: 1669669340, Total: 150, Dropped: 0, Rate: 25.00, Max_rate: 10.00
[2022/11/28 21:02:30] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input1, last_msg_time: 1669669341, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:02:30] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input2, last_msg_time: 1669669341, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:02:30] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input3, last_msg_time: 1669669341, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:02:35] [ info] [filter:throttle_tag:throttle_tag.0] Tag: __global_window__, last_msg_time: 1669669340, Total: 150, Dropped: 0, Rate: 25.00, Max_rate: 10.00
[2022/11/28 21:02:35] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input1, last_msg_time: 1669669341, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:02:35] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input2, last_msg_time: 1669669341, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:02:35] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input3, last_msg_time: 1669669341, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:02:40] [ info] [filter:throttle_tag:throttle_tag.0] Tag: __global_window__, last_msg_time: 1669669355, Total: 299, Dropped: 0, Rate: 49.83, Max_rate: 10.00
[2022/11/28 21:02:40] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input1, last_msg_time: 1669669355, Total: 100, Dropped: 0, Rate: 16.67, Max_rate: 10.00
[2022/11/28 21:02:40] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input2, last_msg_time: 1669669355, Total: 100, Dropped: 0, Rate: 16.67, Max_rate: 10.00
[2022/11/28 21:02:40] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input3, last_msg_time: 1669669355, Total: 99, Dropped: 1, Rate: 16.50, Max_rate: 10.00
[2022/11/28 21:02:45] [ info] [filter:throttle_tag:throttle_tag.0] Tag: __global_window__, last_msg_time: 1669669355, Total: 299, Dropped: 0, Rate: 49.83, Max_rate: 10.00
[2022/11/28 21:02:45] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input1, last_msg_time: 1669669355, Total: 100, Dropped: 0, Rate: 16.67, Max_rate: 10.00
[2022/11/28 21:02:45] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input2, last_msg_time: 1669669355, Total: 100, Dropped: 0, Rate: 16.67, Max_rate: 10.00
[2022/11/28 21:02:45] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input3, last_msg_time: 1669669355, Total: 99, Dropped: 1, Rate: 16.50, Max_rate: 10.00
[2022/11/28 21:02:50] [ info] [filter:throttle_tag:throttle_tag.0] Tag: __global_window__, last_msg_time: 1669669355, Total: 149, Dropped: 0, Rate: 24.83, Max_rate: 10.00
[2022/11/28 21:02:50] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input1, last_msg_time: 1669669355, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:02:50] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input2, last_msg_time: 1669669355, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:02:50] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input3, last_msg_time: 1669669355, Total: 49, Dropped: 1, Rate: 8.17, Max_rate: 10.00
[2022/11/28 21:02:55] [ info] [filter:throttle_tag:throttle_tag.0] Tag: __global_window__, last_msg_time: 1669669355, Total: 149, Dropped: 0, Rate: 24.83, Max_rate: 10.00
[2022/11/28 21:02:55] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input1, last_msg_time: 1669669355, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:02:55] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input2, last_msg_time: 1669669355, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:02:55] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input3, last_msg_time: 1669669355, Total: 49, Dropped: 1, Rate: 8.17, Max_rate: 10.00
[2022/11/28 21:03:00] [ info] [filter:throttle_tag:throttle_tag.0] Tag: __global_window__, last_msg_time: 1669669355, Total: 149, Dropped: 0, Rate: 24.83, Max_rate: 10.00
[2022/11/28 21:03:00] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input1, last_msg_time: 1669669355, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:03:00] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input2, last_msg_time: 1669669355, Total: 50, Dropped: 0, Rate: 8.33, Max_rate: 10.00
[2022/11/28 21:03:00] [ info] [filter:throttle_tag:throttle_tag.0] Tag: tail.var.tmp.input3, last_msg_time: 1669669355, Total: 49, Dropped: 1, Rate: 8.17, Max_rate: 10.00
[2022/11/28 21:03:40] [ info] [filter_throttle_tag] Window "tail.var.tmp.input3" was deleted, msg dropped: 1. CT1669669420   TT1669669360   T1669669355
[2022/11/28 21:03:40] [ info] [filter_throttle_tag] Window "tail.var.tmp.input1" was deleted, msg dropped: 0. CT1669669420   TT1669669360   T1669669355
[2022/11/28 21:03:40] [ info] [filter_throttle_tag] Window "tail.var.tmp.input2" was deleted, msg dropped: 0. CT1669669420   TT1669669360   T1669669355

valgrind --leak-check=full --show-leak-kinds=all /fluent-bit/bin/fluent-bit -c f.conf

==587== LEAK SUMMARY:
==587==    definitely lost: 0 bytes in 0 blocks
==587==    indirectly lost: 0 bytes in 0 blocks
==587==      possibly lost: 0 bytes in 0 blocks
==587==    still reachable: 93,218 bytes in 695 blocks
==587==         suppressed: 0 bytes in 0 blocks
==587==
==587== Use --track-origins=yes to see where uninitialised values come from
==587== For lists of detected and suppressed errors, rerun with: -s
==587== ERROR SUMMARY: 6 errors from 1 contexts (suppressed: 0 from 0)

Signed-off-by: Michael Voelker [email protected]

#5835

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

  • [x] Example configuration file for the change
  • [] Debug log output from testing the change
  • [x ] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [ ] Attached local packaging test output showing all targets (including any new ones) build.

Documentation

  • [x] Documentation required for this feature

Backporting

  • [ ] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

novegit avatar Nov 28 '22 21:11 novegit

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

github-actions[bot] avatar Feb 27 '23 02:02 github-actions[bot]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

github-actions[bot] avatar Nov 27 '24 02:11 github-actions[bot]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

github-actions[bot] avatar Aug 28 '25 02:08 github-actions[bot]